Serving up objects for $0

Mar 18, 2020

This is a blog post on a web service I’ve wanted for myself for a while. It’s an object storage. An alternative to AWS’ S3. I’ve created E3 (Evan’s S3), which can be found here.

Screenshot from 2020-03-18 13-40-54

Many web projects of mine have wanted an object storage -but in keeping things cheap I forgo whatever feature(s) required one. No more. When I want to add file support to smscp I’ll no longer have to live without (experimental file support is enabled there -but just for me).

Making an object storage for free.

Yes, I’m spending a whole $0 on E3 at the moment -but it does have limitations. I only have a handful of terabytes of storage space for it at the moment -and the max file size is 4GB -more than enough for my needs -but not something you can give to the world. Not a concern for me as the number one priority when developing a web service that allows users to upload arbitrary content should be moderation. Otherwise you might find yourself answering to your government when someone tries to push the envelope.

What kind of tech keeps this free? Hosting?

I use to self host Git at my mother’s house on a spare laptop. I moved my things over to sourcehut so I’ve had a spare server. Let’s use it!

The dirty details come down to using MySQL’s LONGBLOB while throwing Memcache in front to keep it fast. Yes, the 4GB restriction is due to LONGBLOB -and I’m currently storing gigantic things in Memcache. On a single server this wouldn’t scale for more than a few power users such as myself.

Technical details.

Well, you have some. E3 makes use of MySQL and Memcache. But you’re just hosting this on a laptop. What do those deploys look like? I haven’t automated the setup yet (blowing up the server), but am looking into Chef, Ansible, and friends for the best way to do that. I’ve thrown NGINX in front of E3’s Go server. The Go server is managed by systemd. When I want to deploy the process is so:

tar czf /tmp/e3.tar.gz e3
scp /tmp/e3.tar.gz alice@mom:~
ssh alice@mom
tar xf e3.tar.gz
cd e3
make
sudo service e3 restart
^D

Pretty old school, huh? Only worrying about a single server instance is nice. If you want to see the systemd service file and NGINX file you can find both here. They’re a little out of date. The current NGINX file is more minimal and the systemd service file has more environment variables now.

Want in?

You’ll have to contact me. I’ll let you figure out how. Once you buy me a beer and/or pizza I’ll create an invite link for you to use. It’s a one time use invite link that will allow you to create an account on E3 tied to a username and password of your choosing -neither of which can be changed at the moment (remember -this is a service built for my needs to be integrated into other projects -not for end users!). You also have to promise me not to do anything illegal. I will check up on user uploads. :-)

Why would I want to use E3?

I use E3 both as an alternative to AWS’ S3 for web service uses and as an alternative to Dropbox for transferring large files between my many devices. I have a Fish shell function/alias that allows me to upload seamlessly.

function upload_file
  set USERNAME evan
  set PASSWORD (pass show e3_evan)
  set FILE (readlink -f $argv[1])
  curl -Ss -u "$USERNAME:$PASSWORD" -F "access=public" -F "data=@$FILE" https://e3.evanjon.es/api
end

alias upload='upload_file $args'

You would no longer be reliant on large tech companies and instead be reliant on the internet connection at my mother’s house. Should you use it for production use? Only if you’re the idiot writing this blog post.

That’s all for now. E3’s been fun to make -and I’m very happy I’m a bit less reliant on large tech companies.