Ship Grip - FIM
File Integrity Monitor
Currently we have a binary available for Linux HERE . This has been built and tested on Ubuntu 18.04 (64-bit). We plan to have binaries for other platforms soon. Windows, OSX, Solaris, and BSD versions are on the way.
You can grab the source for Ship Grip FIM on Git Hub
What it does currently (v0.3.0):
- Checksum every file a directory / filesystem, recursively
- Store checksums in MongoDB database as reports
- Compare reports and report changed missing files
- File checksumming runs in parallel
- Written in Golang!
Coming up for v1.0.0:
- GUI - web and fat client
- Damon mode with scheduled jobs to compare changes
- Alerts and email alerts
- MySQL support in case you don't like MongoDB
- Performance improvements
Why Does It Exist?
The main point of this tool is to make absolutely sure that you know when your files are modified. You may want to do this for several different reasons, for example:
- Protect against unauthorized changes
- Protect against malware
- Protect against bit rot
- Protect against accidental delation modification
Traditionally file integrity monitor tools have been geared more towards security. While security concerns present valid use cases the real driving force behind this tool is to protect against bit rot. Beyond that, I mostly want to know for sure that my files are all intact and not changing. I don't want to have to wonder, I want to know.
Bit rot is a real threat to your data. It occurs when random bits are flipped on disk due to hardware issues. These don't necessarily cause corruption of the file system structure but they may corrupt the actual data of a file. This means that they can't be detected unless you happen to notice that something changed. A single flipped bit can cause huge ammounts of damage. Consider pictures and music files. A single flipped bit in an image file can cause all sorts of distortion. Music files may develop annoying blips. You can read more about bit rot HERE
Doesn't RAID solve this problem?
No, not really, it doesn't. RAID isn't meant to dectect this type of problem and won't be able to determine which disk has the valid bit.
But I Have Backups!
If you can't detect when things are modified you will not have any idea when those backups are needed unless something breaks. Your files could be silently corrupted while you just sit there in ignorance. The corrupted files will then be backed up. As older backups are removed you will eventually be left with nothing but the bad data.
ZFS, BTRFS, etc.
Another solution would be to use a next generation file system such as ZFS or BTRFS. These will detect and automatically fix bit rot. As great as they are, next generation file systems won't detect tampering, malware, or accidents.
Speed
When auditing a large filesystem with many GB or TB of data, a single scan can take a really long time. The slower the can, the harder it is to have it scheduled on a regular basis. If a single scan of a large volume takes longer than 24 hours, it can't really be run everyday. Speed is obviously a critical factor.
When designing Ship Grip - FIM we took this into consideration. We decided that it needed to be able to run multiple concurrent routines simultaneously. This is part of the reason we chose to use golang. We wanted to focus on speed and parallel execution. Golang is great for concurrency and seemed like the perfect language for the job. So far I'm happy with the results.