Index

s3-bsync / bb1358b

Bidirectional syncing tool to sync local filesystem directories with S3 buckets. (Incomplete)

Latest Commit

{#}TimeHashSubjectAuthor#(+)(-)GPG?
118 Oct 2021 11:09bb1358bInitial commitJosh Stockin11020G

Blob @ s3-bsync / README.html

text/html4740 bytesdownload raw
1<style>body{max-width:650px;margin:20px auto;color:#252525}</style>
2<h1>s3-bsync</h1>
3<p>Bidirectional syncing tool to sync local filesystem directories with S3
4buckets. Written by <a href="https://joshstock.in">Josh Stockin</a>.</p>
5<h3>Behavior</h3>
6<p>After an initial sync (manually handling conflicts and uncommon files), the S3
7bucket maintains precedence. Files with the same size and modify time on both
8hosts are ignored. A newer copy of a file always overwrites the corresponding
9old, regardless of changes in the old. (In other words, <strong>there is no manual
10conflict resolution after first sync. Conflicting files are handled
11automatically as described here.</strong> This script is meant to run without input
12or output by default, in a cron job for example.) Untracked files, in either
13S3 or on the local machine, are copied to the opposite host and tracked.
14Tracked files that are moved or removed on either host are moved or removed on
15the corresponding host, with the tracking adjusted accordingly. Ultimately,
16after a sync, the <code>.state.s3sync</code> state tracking file should match the contents
17of the S3 bucket's synced directories.</p>
18<h3>Installation</h3>
19<p>Depends on <code>python3</code> and <code>aws-cli</code>. Both can be installed with your package
20manager.</p>
21<p>Install with <code>python3 setup.py install</code>. Root permissions not required.
22<em>This program does not manage S3 authentication or <code>aws-cli</code> credentials. You
23must do this yourself with the <code>aws configure</code> command, or other means of
24IAM/S3 policy.</em></p>
25<h4>Source files</h4>
26<p><code>setup.py</code> manages installation metadata.</p>
27<h4>Created files and .s3syncignore</h4>
28<p>The default file used to store sync information is <code>~/.state.s3sync</code>, but this
29location can be reconfigured. The file uses the binary s3sync file format
30specified later in this document. If you want to intentionally ignore
31untracked files, use a <code>.s3syncignore</code> file, in the same manner as
32<a href="https://git-scm.com/docs/gitignore"><code>.gitignore</code></a>.</p>
33<h2>s3sync file format</h2>
34<p>The <code>.state.s3sync</code> file saved in home directory defines the state of tracked
35objects from the specified S3 buckets and key prefixes used in the last sync.</p>
36<h3>Control bytes</h3>
37<pre><code>90 - Begin bucket block
3891 - End bucket block
3992 - Begin directory map
4093 - End directory map
4194 - Begin object block
4295 - End object block
4396 - ETag type MD5
4497 - ETag type null-terminated string (non-MD5)
4598
4699
479A - Begin metadata block
489B - End metadata block
499C
509D - File signature byte
519E
529F - File signature byte
53</code></pre>
54<h3>File structure</h3>
55<pre><code>Header {
56 File signature - 4 bytes - 9D 9F 53 33
57 File version - 1 byte - 01
58}
59
60Metadata block {
61 Begin metadata block control byte - 9A
62 Last synced time - 8 bytes uint
63 End metadata block control byte - 9B
64}
65
66Bucket block {
67 Begin bucket block control byte - 90
68 Bucket name - null-terminated string
69 Directory map {
70 Begin directory map block control byte - 92
71 Path to local directory - null-terminated string
72 S3 key prefix - null-terminated string
73 Recursive sync - 1 byte boolean
74 End directory map block control byte - 93
75 }...
76 Recorded object {
77 Begin object block control byte - 94
78 Key - null-terminated string
79 Last modified time - 8 bytes uint
80 ETag type - 96 or 97
81 ETag - 16 bytes or null-terminated string
82 File size - 8 bytes uint
83 End object block control byte - 95
84 }...
85 End bucket block control byte - 91
86}...
87</code></pre>
88<h2>Copyright</h2>
89<p>This program is copyrighted by <a href="https://joshstock.in/">Joshua Stockin</a> and
90licensed under the <a href="LICENSE">MIT License</a>.</p>
91<p>A form of the following should be present in each source file.</p>
92<pre><code class="language-txt">s3-bsync Copyright (c) 2021 Joshua Stockin
93&lt;https://joshstock.in&gt;
94&lt;https://git.joshstock.in/s3-bsync&gt;
95
96This software is licensed and distributed under the terms of the MIT License.
97See the MIT License in the LICENSE file of this project's root folder.
98
99This comment block and its contents, including this disclaimer, MUST be
100preserved in all copies or distributions of this software's source.
101</code></pre>
102<p>&lt;<a href="https://joshstock.in">https://joshstock.in</a>&gt; | <a href="mailto:josh@joshstock.in">josh@joshstock.in</a> | joshuas3#9641</p>
103