cyberes
/
crazy-file-server
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity
A heavy-duty web file browser for cRaZy files.
61 Commits 1 Branch 3 Tags 355 KiB
Go 99.5%
Shell 0.5%
Go to file
Cyberes 933670c63a adjust json response, minor clean up 2024-03-23 17:38:02 -06:00
Doc Merge dev to master 2023-12-08 22:25:59 -07:00
src adjust json response, minor clean up 2024-03-23 17:38:02 -06:00
.gitignore rename all modules to lowercase 2024-03-17 12:22:02 -06:00
LICENSE Initial commit 2023-07-13 13:54:42 -06:00
README.md update readme 2024-03-23 17:37:16 -06:00
build.sh build script cleans dir 2024-03-17 20:49:49 -06:00
config.yml.sample rename config variable 2024-03-17 20:33:32 -06:00
todo.txt redo all logging messages, minor qol 2024-03-15 22:20:10 -06:00

README.md

crazy-file-server

A heavy-duty web file browser for cRaZy files.

The whole schtick of this program is that it caches the file structure of your dataset so that the server doesn't have to do I/O operations on every request. By doing the processing upfront when the server starts, we can keep requests snappy and responsive.

I needed to serve a very large dataset full of small files publicly over the internet in an easy to browse website. The existing solutions were subpar and I found myself having to create confusing Openresty configs with complex CDN caching to keep things responsive and server load low. I gave up and decided to build my own solution from the ground up.

System Setup

You'll need at least 5GB of RAM CrazyFS is heavily threaded, so you'll want at least an 8-core machine.

You absolutely need an SSD for this. With two SSDs in a RAID1 ZPOOL, my server was able to crawl over 7 million files stored in a very complicated directory tree in under 3 minutes.

It's also possible to use a very fast SSD as swap in case your dataset needs more memory than you have in RAM. The Samsung 980PRO NVMe drive worked very well for me as a swap drive.

You'll need something like Nginx if you want SSL. Also, crazyfs works best with an HTTP cache in front of it. I set my CDN to cache responses for 2 hours.

This program was designed to run on a Linux machine. Not sure if this works on Windows.

Features

  • Automated cache management. Fill the cache when the starts, or as requests come in.
  • Front end agnostic design.
  • Elasticsearch integration.
  • File browsing API.
  • Download API.
  • Admin API.

Install

Download the binary or install Go and build it via ./build.sh.

Use

  1. Edit config.yml. It's well commented.
  2. ./crazyfs --config /path/to/config.yml. You can use -d for debug mode to see what it's doing.

By default, it looks for your config in the same directory as the executable: ./config.yml or ./config.yaml.

Native searching is included with the server, but it doesn't work very well and is slow. You'll probably want to set up Elasticsearch.