How to Save Software Deployment Time
Azoft Blog How to Save Software Deployment Time

How to Save Software Deployment Time

By Anton Demenev on April 8, 2015

Step-by-step instructions for system administrators and DevOps engineers

How to Save Software Deployment Time

Disk subsystem is a bottleneck for most servers, except for large datacentres with SAN storages based on SSD. Every disk IO operation takes a few milliseconds, while CPU and RAM perform millions of operations during the same period. But there’s a simple way for development teams to deploy large datasets that consist of small files. The following tips will help you save time significantly, making the whole process dozens of times faster.

Test storage server characteristics

Our test system has three Intel i3 cores and 4GB of RAM memory. It is still stable and works comfortably even if these parameters are reduced to 2GB of RAM and 2 processor cores.

What about our testing environment?

All the test machines are based on Debian7, hypervisor host is based on Windows7. In real environment you can use any hypervisor you like.
Dataset consists of more than 2 million small files and weighs about 5GB with no directory tree based optimisation. Tar packaging without compression takes about 4 hours considering minimal disk usage priority.

how to save deployment time environment

The development dataset is based on backups from the Production server. It’s a massive of real data, which consist of daily full database backup, daily files delta backup and weekly full files backup. All the data is packed in tarball on the Production server and uploaded to the storage server.

how to save deployment time dataset lifecycle

As soon as data is unpacked, we get a Big Directory. This process lasts about half an hour, and full directory listing is even slower. To copy tarball you need about 10 minutes and about an hour to copy the data unpacked.

In this case, dataset deployment for 4 developers takes 40 minutes on each computer, working at the highest IO loading speed. If the developer damages his data, the whole process starts again. Using this model you spend a lot of precious developers’ time.

The main idea is to make deploy process quick, simple and cheap.

Our deploy scheme is based on Oracle ZFS filesystem. It has a lot of kill features, such as:

  • Read-only snapshots wiсh reduce storage space loss,
  • Read-write clones,
  • Ease of use,
  • Failure tolerance, built-in RAID analog from the box. RAIDZ technology, make working without RAID controllers and software RAID subsystems being installed.

You can find tips on how to install this environment on ZFS on Linux homepage.

So, let the deployment begin!

1. Add repository, install ZFS utilities and create the storage

To install ZFS on Debian you can use ZoL repo.

$ su -
# wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_5_all.deb
# dpkg -i zfsonlinux_5_all.deb
# wget http://zfsonlinux.org/4D5843EA.asc -O - | apt-key add -
# apt-get update
# apt-get install debian-zfs

You need three virtual hard drives to create RAIDZ array and one virtual hard drive for your OS. Our system is installed on /dev/sdd drive.

root@debian-7-64-storage:/# parted -l
Model: VBOX HARDDISK (scsi)
Disk /dev/sda: 51.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number  Start   End 	Size	File system  Name 	Flags
 1  	2097kB  51.5GB  51.5GB  zfs      	primary
Model: VBOX HARDDISK (scsi)
Disk /dev/sdb: 51.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number  Start   End 	Size	File system  Name 	Flags
 1  	2097kB  51.5GB  51.5GB  zfs      	primary
Model: VBOX HARDDISK (scsi)
Disk /dev/sdc: 51.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number  Start   End 	Size	File system  Name 	Flags
 1  	2097kB  51.5GB  51.5GB  zfs      	primary
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdd: 51.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start   End 	Size	Type  	File system 	Flags
 1  	1049kB  49.4GB  49.4GB  primary   ext4        	boot
 2  	49.4GB  51.5GB  2137MB  extended
 5  	49.4GB  51.5GB  2137MB  logical   linux-swap(v1)

Create the storage space:

root@debian-7-64-storage:~# zpool create main-storage raidz /dev/sda1 /dev/sdb1 /dev/sdc1
root@debian-7-64-storage:~# zpool list
NAME       	SIZE  ALLOC   FREE	CAP  DEDUP  HEALTH  ALTROOT
main-storage   143G   232K   143G 	0%  1.00x  ONLINE  -
root@debian-7-64-storage:~# zfs create main-storage/dev-storage -o mountpoint=/dev-storage
root@debian-7-64-storage:~# zfs list
NAME                   	USED  AVAIL  REFER  MOUNTPOINT
main-storage           	195K  93.8G  38.6K  /main-storage
main-storage/dev-storage  38.6K  93.8G  38.6K  /dev-storage

2. Deploy dataset initial test

Extract the initial copy of your data:

root@debian-7-64-storage:/dev-storage# time tar -xvf ./dataset.tar | wc -l
1038756
real	29m27.212s
user	5m17.373s
sys 	3m25.046s

Once you’re finished, make a few copies:

root@debian-7-64-storage:/# ls -1l /dev-storage/
total 14
...
drwxr-xr-x 6 root root 6 Mar 27 19:19 stage1
drwxr-xr-x 6 root root 6 Mar 27 19:19 stage2

At this moment each directory contains a lot of small files - approximately 2 million in total.

3. Deploy developers’ datasets

Create the initial snapshot.

root@debian-7-64-storage:~# zfs snapshot main-storage/dev-storage@initial
root@debian-7-64-storage:~# zfs list -t snapshot
NAME                           	USED  AVAIL  REFER  MOUNTPOINT
main-storage/dev-storage@initial  	0  	-  8.37G  -

This snapshot is a read-only copy of your data. To change it, clone the initial snapshot.

root@debian-7-64-storage:~# mkdir -p /snapshots/dev-{1,2}
root@debian-7-64-storage:~# zfs clone main-storage/dev-storage@initial main-storage/dev-1 -o mountpoint=/snapshots/dev-1
root@debian-7-64-storage:~# zfs clone main-storage/dev-storage@initial main-storage/dev-2 -o mountpoint=/snapshots/dev-2
root@debian-7-64-storage:~# zfs list
NAME                   	USED  AVAIL  REFER  MOUNTPOINT
main-storage          	8.37G  85.4G  38.6K  /main-storage
main-storage/dev-1     	128K  85.4G  8.37G  /snapshots/dev-1
main-storage/dev-2    	1.33K  85.4G  8.37G  /snapshots/dev-2
main-storage/dev-storage  8.37G  85.4G  8.37G  /dev-storage

Thereafter, take the clones’ snapshots. This way you make data reset possible in the case of damage.

root@debian-7-64-storage:~# zfs snapshot main-storage/dev-1@initial
root@debian-7-64-storage:~# zfs snapshot main-storage/dev-2@initial
root@debian-7-64-storage:~# zfs list -t snapshot
NAME                           	USED  AVAIL  REFER  MOUNTPOINT
main-storage/dev-1@initial    	1.33K  	-  8.37G  -
main-storage/dev-2@initial        	0  	-  8.37G  -
main-storage/dev-storage@initial   131K  	-  8.37G  -

We couldn’t set quota restrictions for our test storage, but it is possible if you use a setting attribute “quota=<quotasize>” by zfs utility.

4. Deploy data to developers sandboxes

Finally, you have two mounted virtual filesystems, which means you can export the writable copy to developer’s machine.

root@debian-7-64-storage:~# mount
…..
main-storage on /main-storage type zfs (rw,relatime,xattr,noacl)
main-storage/dev-storage on /dev-storage type zfs (rw,relatime,xattr,noacl)
main-storage/dev-1 on /snapshots/dev-1 type zfs (rw,relatime,xattr,noacl)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
main-storage/dev-2 on /snapshots/dev-2 type zfs (rw,relatime,xattr,noacl)

We’ve chosen NFS to transfer data between storage and developer’s computer. It is simple, fast enough and requires minimum of the system resources.

root@debian-7-64-storage:~# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
	link/ether 08:00:27:a3:f7:15 brd ff:ff:ff:ff:ff:ff
	inet 172.16.0.111/16 brd 172.16.255.255 scope global eth0
root@debian-7-64-dev:~# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
	link/ether 08:00:27:e9:e5:ba brd ff:ff:ff:ff:ff:ff
	inet 172.16.0.105/16 brd 172.16.255.255 scope global eth0

You can activate Jumbo Frames to improve the performance:

root@debian-7-64-storage:~# ifconfig eth0 mtu 9000
root@debian-7-64-dev:~# ifconfig eth0 mtu 9000
root@debian-7-64-storage:~# netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR	TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   	9000 0   2367100  	0  	0 0   	2702231  	0  	0  	0 BMRU

Check its work.

root@debian-7-64-storage:~# ping 172.16.0.105 -M do -s 8972
PING 172.16.0.105 (172.16.0.105) 8972(9000) bytes of data.
8980 bytes from 172.16.0.105: icmp_req=1 ttl=64 time=0.611 ms
8980 bytes from 172.16.0.105: icmp_req=2 ttl=64 time=0.413 ms
8980 bytes from 172.16.0.105: icmp_req=3 ttl=64 time=0.284 ms

Set up the sharing. To simplify the process I haven’t adjusted the additional authentication and access rights subsystems. Log in at the host address.

root@debian-7-64-storage:~# cat /etc/exports
/snapshots/dev-1           	172.16.0.105/32(rw,nohide,insecure,no_subtree_check,async,no_root_squash,fsid=0)
/snapshots/dev-2           	172.16.0.62/32(rw,nohide,insecure,no_subtree_check,async,no_root_squash,fsid=0)

Launch service.

root@debian-7-64-storage:~# service nfs-kernel-server start
[ ok ] Exporting directories for NFS kernel daemon....
[ ok ] Starting NFS kernel daemon: nfsd mountd.

Mount new filesystem to the developer’s machine. I disable metadata files and increase packet size for better performance.

root@debian-7-64-dev:~# mount -t nfs 172.16.0.111:/snapshots/dev-1 /srv/data -o noacl,nocto,noatime,nodiratime,rsize=8192,wsize=8192
root@debian-7-64-dev:~# mount
172.16.0.111:/snapshots/dev-1 on /srv/data type nfs (rw,noatime,nodiratime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,nocto,noacl,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.111,mountvers=3,mountport=42230,mountproto=udp,local_lock=none,addr=172.16.0.111)

At this stage, the exported directories are available:

root@debian-7-64-dev:/srv/data# ls -1l
total 14
drwxr-xr-x 6 root root 6 Mar 27 20:19 stage1
drwxr-xr-x 6 root root 6 Mar 27 20:19 stage2

Repeat the same actions for the rest of developer’s computers. If you don’t want to use the command line, you can write a simple helper script for developers’ computers that provides all the dataset management functions via ssh.

Following this scheme, you will definitely save loads of time for your team and become a deployment guru.

VN:F [1.9.22_1171]
Rating: 4.5/5 (2 votes cast)
VN:F [1.9.22_1171]
Rating: +2 (from 2 votes)
How to Save Software Deployment Time, 4.5 out of 5 based on 2 ratings



Request a Free Quote
 
 
 

Please enter the result and submit the form

Content created by Anton Demenev