XipZ
Mini packer ▶►▸ for small programs.
Classes | Macros | Typedefs | Functions
xipz.cc File Reference

Source file containing the main function. More...

#include <inttypes.h>
#include <iostream>
#include <fstream>
#include <vector>
#include <iterator>
#include <stdexcept>
#include <algorithm>
#include <numeric>
#include <sstream>
#include <boost/format.hpp>
#include "cmdline.h"
#include "data.hh"
#include "qadz.hh"
#include "decrunchxipzstub.inc"
Include dependency graph for xipz.cc:

Go to the source code of this file.

Classes

struct  Bits
 
class  HistEntry
 Histogram entry. More...
 

Macros

#define POS_OF_END_OF_CDATA   0x38
 Position in the stub where the end of the compressed data is stored.
 
#define POS_OF_BEGIN_OF_CDATA   0x3d
 Position in the stub where the beginning of the compressed data is stored.
 
#define POS_OF_N   0x46
 Position in the stub where the end of the number of bits for the compression is stored.
 
#define POS_OF_DEST   0x6f
 Position in the stub where the address is stored at which the data is decompressed.
 
#define POS_OF_JMP   0x5c
 Position in the stub where the address is stored at which the final JMP is performed.
 
#define POS_OF_STOPREADING   0x58
 Position in the stub where the high byte of the destination address is stored at which the decompression process stops.
 

Typedefs

typedef std::vector< HistEntryHistorgramArray
 Convenience type for a histogram array. More...
 
typedef std::array< Bits, 256 > CompressionBits
 Convenience type definition for an array of 256 byte counts.
 

Functions

std::ostream & operator<< (std::ostream &o, const Bits &x)
 Output operator for Bits. More...
 
Data read_data (const std::string &fname)
 Read data from a file. More...
 
HistorgramArray calc_histo (const Data &data)
 calculate the histogram More...
 
std::ostream & operator<< (std::ostream &o, const HistEntry &x)
 Output a histogram entry. More...
 
float calc_comp (int n, const Data &data, const HistorgramArray &histarr)
 calculate compressed bytes More...
 
CompressionBits create_compression_bits (const HistorgramArray &compressable, int n)
 create compression table More...
 
std::ostream & write_stub (std::ostream &out, int n, uint16_t size, uint16_t loadaddr, uint16_t jmp)
 write the decrunch stub More...
 
std::ostream & write_compression_table (std::ostream &out, int n, const HistorgramArray &histe)
 write compression table More...
 
std::ostream & write_compressed_data (std::ostream &out, const std::vector< uint8_t > &data)
 write the compressed data More...
 
std::vector< uint8_t > create_compressed_data (const Data &data, const CompressionBits &compbits)
 Create compressed data. More...
 
void output_64_common (const HistorgramArray &shisto)
 
int choose_optimal_n (const Data &data, const HistorgramArray &shisto)
 choose optimal number of bits More...
 
int main_xipz (const std::string &inputname, const std::string &outputname, bool raw, int jump)
 main function using xip More...
 
int main_qadz (const std::string &inputname, const std::string &outputname, bool raw, int jump)
 main function using qadz More...
 
int main (int argc, char **argv)
 main function using xip More...
 

Detailed Description

Source file containing the main function.

Currently contains the main function of the program and the xipz compression algorithm.

Definition in file xipz.cc.

Typedef Documentation

◆ HistorgramArray

typedef std::vector<HistEntry> HistorgramArray

Convenience type for a histogram array.

Used to store the HistEntry entries. Mostly this table is sorted so that the most common entries are first.

Definition at line 109 of file xipz.cc.

Function Documentation

◆ calc_comp()

float calc_comp ( int  n,
const Data data,
const HistorgramArray histarr 
)

calculate compressed bytes

Calculate the number of compressed bytes at a given n.

Parameters
nnumber of bits
datafile data
histarrsorted histogram data
Returns
number of compressed bytes (with fractional part)

Definition at line 199 of file xipz.cc.

◆ calc_histo()

HistorgramArray calc_histo ( const Data data)

calculate the histogram

For each byte in the input data the frequency is calculated.

The histogram entries are sorted according to their frequencies. The most common bytes are moved to teh beginning of the array.

Parameters
databinary data
Returns
vector of histogram entries, sorted

Definition at line 158 of file xipz.cc.

◆ choose_optimal_n()

int choose_optimal_n ( const Data data,
const HistorgramArray shisto 
)

choose optimal number of bits

Calculate the compressed data for each number of bytes and return the optimal number of bits.

Parameters
datafile data
shistosortet histogram aka frequencies
Returns
optimal number of bits

Definition at line 400 of file xipz.cc.

◆ create_compressed_data()

std::vector<uint8_t> create_compressed_data ( const Data data,
const CompressionBits compbits 
)

Create compressed data.

The table of compression bits (CompressionBits) is all that is needed to compress the data in data. Every output token is prepended with a '1' bit if it is a literal token (aka byte) or with a '0' bit if it is compressed data. In this case only the bits in the compbits table are written.

If not a whole byte is filled at the end, then zero bits are appended.

Parameters
databinary data to crunch
compbitscompression table
Returns
compressed data

Definition at line 333 of file xipz.cc.

◆ create_compression_bits()

CompressionBits create_compression_bits ( const HistorgramArray compressable,
int  n 
)

create compression table

Generates a table of 256 entries where the n bits are stored according to their frequency or the eight bits of the byte value if not compressed.

Parameters
compressablehistogram entries for the generation of the compression table
nhow many bits to use
Returns
Table of 256 entries containing the bits for each byte, type is CompressionBits

Definition at line 225 of file xipz.cc.

◆ main()

int main ( int  argc,
char **  argv 
)

main function using xip

Main function which takes a single file name as an argument.

Definition at line 482 of file xipz.cc.

◆ main_qadz()

int main_qadz ( const std::string &  inputname,
const std::string &  outputname,
bool  raw,
int  jump 
)

main function using qadz

Main function for compression using the qadz (LZ77-like) algorithm.

Parameters
inputnameinput filename
outputnameoutout filename
rawshould the compressed data be written raw (without decompression stub)
jumpjump address, -1 = equal to load address

Definition at line 457 of file xipz.cc.

◆ main_xipz()

int main_xipz ( const std::string &  inputname,
const std::string &  outputname,
bool  raw,
int  jump 
)

main function using xip

Main function for compression using the xip algorithm.

Parameters
inputnameinput filename
outputnameoutout filename
rawshould the compressed data be written raw (without decompression stub)
jumpjump address, -1 = equal to load address

Definition at line 425 of file xipz.cc.

◆ operator<<() [1/2]

std::ostream& operator<< ( std::ostream &  o,
const Bits x 
)

Output operator for Bits.

This function will print the bits stored in a human-readable manner.

Definition at line 80 of file xipz.cc.

◆ operator<<() [2/2]

std::ostream& operator<< ( std::ostream &  o,
const HistEntry x 
)

Output a histogram entry.

This is used to display the nice table of the 64 most common bytes.

Parameters
ooutput stream to write to
xHistEntry to output
Returns
output stream for stl-conforming usage

Definition at line 184 of file xipz.cc.

◆ output_64_common()

void output_64_common ( const HistorgramArray shisto)

Output the most common bytes and their frequencies.

Parameters
shistosorted histogram array

Definition at line 375 of file xipz.cc.

◆ read_data()

Data read_data ( const std::string &  fname)

Read data from a file.

Input is read and an exception is thrown if the file can not be opened.

Parameters
fnamefile name
Returns
Data object with loaded binary data

Definition at line 123 of file xipz.cc.

◆ write_compressed_data()

std::ostream& write_compressed_data ( std::ostream &  out,
const std::vector< uint8_t > &  data 
)

write the compressed data

Write the binary compressed data into the output stream.

Parameters
outoutput stream to write to
databinary data to write
Returns
output stream

Definition at line 312 of file xipz.cc.

◆ write_compression_table()

std::ostream& write_compression_table ( std::ostream &  out,
int  n,
const HistorgramArray histe 
)

write compression table

This table contains the 2^n most common bytes in order of decreasing frequency.

Parameters
outoutput stream to write to
nnumber of bits
histehistogram entries, they contain the sorted list of bytes
Returns
output stream

Definition at line 295 of file xipz.cc.

◆ write_stub()

std::ostream& write_stub ( std::ostream &  out,
int  n,
uint16_t  size,
uint16_t  loadaddr,
uint16_t  jmp 
)

write the decrunch stub

The decrunching stub is written and all parameters like decrunching address and jump address are adjusted. Please take care to adjust all the defines above otherwise the code will probably just crash. It may also fry your cat so be careful!

Parameters
outoutput stream to write to
nnumber of bits
sizenumber of compressed bytes
loadaddroriginal load address
jmpjump address to jump after decrunching
Returns
output stream

Definition at line 253 of file xipz.cc.