This post is now also on the Official Red Hat OpenShift blog!

2020 Update: This post was written for OpenShift 3.x. With the new 4.x version, integration will need to be different. I’ll write a updated post when time allows.

Do you have an OpenShift installation, maybe a test cluster, but no fancy storage solution to provide your Persistent Volumes? Most people would turn to NFS for this, but did you know that it’s almost as easy to set up a simple iSCSI server? This blog post will walk through a simple example.

Why use block storage instead of NFS? Well, first off, it’s not always preferable. NFS (or something like GlusterFS) is required when multiple pods must share the same Persistent Volume. However, for pods that require a high degree of data integrity, like databases, block storage is preferred. NFS has several issues that can compromise data integrity for these kinds of pods. For more information, see Olaf Kirch’s Why NFS Sucks paper.

This setup is not recommended for anything but test/lab environments. It has no security, no high-availability, and no fancy management tool. Use this information at your own risk.

Terminology

iSCSI has its own vocabulary that takes some getting used to:

Target – an iSCSI export. This is what the iSCSI server provides.
Initiator – The client.
IQN – iSCSI Qualified Name. A unique name that identifies iSCSI targets and initiators.
LUN – Logical Unit Number. This represents the actual disk. There can be multiple LUNs exported by a Target.
ACL – Access Control List. Usually a list of allowed initiator IQNs
Portal – a IP and port combination that the target is accessible from.
TPG – Target Portal Group. A grouping of LUNs, ACLs, and Portals that defines access permissions for the Target.

Prerequisites: Deciding on a backstore

In Linux, there are several options for what storage to actually use. In this blog we’ll talk about two kinds: file and block devices. I highly recommend using LVM to manage block devices, but you could also use disk partitions or files.

Files are generally slower, but may be more convenient if you can’t modify the partition table on your iSCSI server, but have free space available.

Nested How-To: Create a large, empty file

To create an arbitrarily-sized empty file for use as an iSCSI back store, use the dd command.

Example of creating a 1024 MB file:

[root@iscsi-server ~]# dd if=/dev/zero of=/path/to/file bs=1M count=1024

dd is a weird command, let’s break down the options:

if stands for “input file”. In this case we’re pulling data from /dev/zero, which is a special device that outputs an endless stream of 0x00 bytes.
of stands for “output file”. This should be the file name you want to create.
bs stands for “block size”. This tells dd how much data to read and write per iteration.
count This tells dd how many times to read/write.

So, bs*count=file size. Therefore, this creates a file of 1024 MB or 1 GB. The block size may influence how much time it takes to create a file. 1 MB is not the optimal write size for most disks. It’s possible to optimize the operation by tweaking the block size, but that’s way out of scope for this post.

Nested How-To: LVM

LVM isn’t strictly required for iSCSI, but it makes the management of block devices much easier. LVM allows block devices to be created on the fly from a pool of available storage. The block devices are called logical volumes, and can be created, deleted and resized on the fly. Logical volumes are contained within volume groups. Volume groups are collections of physical volumes which are presented as a unified pool of storage.

The LVM tools are contained in the lvm2 package, and can be installed via yum install -y lvm2. In the default install of RHEL and CentOS, these tools should be installed.

In this example, we have 100 GB of storaged attached to my my iSCSI server as /dev/vdb.

[root@iscsi-server ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 50G 0 disk
└─vda1 253:1 0 50G 0 part /
vdb 253:16 0 100G 0 disk

First, we have to create a Physical Volume using the pvcreate command. This example uses the whole disk, but if using a partition, be sure that the partition type is 0x8e (Linux LVM).

[root@iscsi-server ~]# pvcreate /dev/vdb
  Physical volume "/dev/vdb" successfully created.
[root@iscsi-server ~]# pvs
  PV         VG Fmt  Attr PSize   PFree  
  /dev/vdb      lvm2 ---  100.00g 100.00g

Next, a volume group, my_vg, needs to be created using the relevant Physical Volumes. Here we only have one physical volume, but a Volume Group can contain many physical volumes and will present all physical volumes associated to the volume group as a single storage pool:

[root@iscsi-server ~]# vgcreate my_vg /dev/vdb
  Volume group "my_vg" successfully created
[root@iscsi-server ~]# vgs
  VG    #PV #LV #SN Attr   VSize   VFree  
  my_vg   1   0   0 wz--n- 100.00g 100.00g

Finally, we can now create storage. The lvcreate command will create a logical volume. There are a whole suite of lv* commands, including lvextend, and lvremove.

[root@iscsi-server ~]# lvcreate --name my_lv -L 1G my_vg
  Logical volume "my_lv" created.
[root@iscsi-server ~]# lvs
  LV    VG    Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  my_lv my_vg -wi-a----- 1.00g                                                    
[root@iscsi-server ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda           253:0    0   50G  0 disk 
└─vda1        253:1    0   50G  0 part /
vdb           253:16   0  100G  0 disk 
└─my_vg-my_lv 252:0    0    1G  0 lvm  

This logical volume is now available for use. The device mapper should present it at /dev/mapper/my_vg-my_lv. If we ever wanted to create more logical volumes, we’d just need to run another lvcreate command.

Now that we have our logical volume, there are setup steps on both the iSCSI server as well as any initiators (clients).

Installing the iSCSI server

On the iSCSI server, installing the targetcli package will install all the required dependencies for the iSCSI server. The target.service unit is the actual service. Port 3260 UDP and TCP must be open.

yum install -y targetcli

firewall-cmd --add-service iscsi-target --permanent && firewall-cmd --reload

Installing the iSCSI client

The iSCSI client needs the iscsi-initiator-utils package. This should already be installed on OpenShift nodes.

Each iSCSI initiator (i.e. the client) is required to have a unique IQN (iSCSI Qualified Name). In RHEL and CentOS, a random one is generated at install time, but you may wish to set more meaningful names. This IQN is used when allowing the client to access iSCSI targets.

Example of a randomly generated IQN:

[root@iscsi-client ~]# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:d1fa8aab755

There is nothing wrong with keeping the default random name. However, it can be helpful to assign meaningful names to each client. After editing the /etc/iscsi/initiatorname.iscsi file, restart iscsid.service for the new InitiatorName to take affect.

If setting a IQN, it is EXTERMELY IMPORTANT that every initiator has a UNIQUE IQN!

Exporting a Target

Now that we have our setup out of the way, it’s time to actually export a target. In RHEL7/CentOS7, this is accomplished with the targetcli command.

targetcli provides an interactive command prompt and presents the iSCSI configuration as a directory tree. Configuring a new target requires a few steps:

Create a backstore object. This will represents the actual block device
Create a target, which will automatically create a target portal group
Assign the backstore to the target portal group
Assign the initiator IQNs to the target portal group’s access control list

Let’s walk through a simple targetcli session:

[root@iscsi-server ~]# targetcli
targetcli shell version 2.1.fb41
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]

Above, we’ve started targetcli and run ls. At the moment, there is nothing configured.

/> cd backstores/block 
/backstores/block> create dev=/dev/mapper/my_vg-my_lv name=my_iscsi_pv
Created block storage object my_iscsi_pv using /dev/mapper/my_vg-my_lv.
/backstores/block> ls
o- block .................................................. [Storage Objects: 1]
  o- my_iscsi_pv ....... [/dev/mapper/my_vg-my_lv (1.0GiB) write-thru activated]

We’ve now created a backstore object by changing my directory to /backstores/block and issuing the create command. The dev parameter is the device name of the logical volume created previously, it could be any unused device (like a hypothetical /dev/sda4), and the name is an arbitrary name of my choosing that we will use in later commands.

For a file instead of a device, we’ll do things slightly differently:

/> cd backstores/fileio 
/backstores/fileio> create file_or_dev=/myfile name=myfile
Created fileio myfile with size 1073741824
/backstores/fileio> ls
o- fileio ................................................. [Storage Objects: 1]
  o- myfile .......................... [/myfile (1.0GiB) write-back deactivated]
/backstores/fileio> 

Notice we changed to a different directory: /backstores/fileio instead of /backstores/block. The options passed to create are also slightly different.

/backstores/block> cd /iscsi
/iscsi> create iqn.2017-04.com.example.mydomain:myiscsipv
Created target iqn.2017-04.com.example.mydomain:myiscsipv.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi> ls
o- iscsi .............................................................................................................. [Targets: 1]
  o- iqn.2017-04.com.example.mydomain:myiscsipv .......................................................................... [TPGs: 1]
    o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
      o- acls ............................................................................................................ [ACLs: 0]
      o- luns ............................................................................................................ [LUNs: 0]
      o- portals ...................................................................................................... [Portals: 1]
        o- 0.0.0.0:3260 ....................................................................................................... [OK]

We’ve now created a target by changing our directory to /iscsi and running the create command. In this example, we specified the IQN of the target, but if an IQN is ommited, the create command will generate a random IQN.

Notice that by creating the target object, a target portal group, tpg1 was also created. A target can have more than one TPG assigned to it, with different LUNs, ACLs, and Portals. In our “quick-and-dirty” setup, we’ll just stick to a single TPG.

/iscsi> cd iqn.2017-04.com.example.mydomain:myiscsipv/tpg1/luns 
/iscsi/iqn.20...ipv/tpg1/luns> create storage_object=/backstores/block/my_iscsi_pv 
Created LUN 0.
/iscsi/iqn.20...ipv/tpg1/luns> ls
o- luns .............................................................. [LUNs: 1]
  o- lun0 ........................ [block/my_iscsi_pv (/dev/mapper/my_vg-my_lv)]

We’ve now assigned a LUN to the target. Again, we changed our directory to the luns directory and used the create command.

A TPG can have multiple LUNs associated with it. To associate another LUN, run the create command again with a different backstore object. In OpenShift, there is an option for LUN number in the iSCSI PersistentVolume options. It’s a matter of taste as to whether to create one target for OpenShift that exports many LUNs, or to have multiple targets with fewer LUNs.

My personal preference is to have different targets for different uses. So, one target with multiple LUNs for general purpose PVs, and a different target for infra-related PVs (e.g. for the PVs related to Aggregate Logging).

/iscsi/iqn.20...ipv/tpg1/luns> cd /iscsi/iqn.2017-04.com.example.mydomain:myiscsipv/tpg1/acls 
/iscsi/iqn.20...ipv/tpg1/acls> create wwn=iqn.1994-05.com.redhat:d1fa8aab755
Created Node ACL for iqn.1994-05.com.redhat:d1fa8aab755
Created mapped LUN 0.
/iscsi/iqn.20...ipv/tpg1/acls> ls
o- acls .............................................................. [ACLs: 1]
  o- iqn.1994-05.com.redhat:d1fa8aab755 ....................... [Mapped LUNs: 1]
    o- mapped_lun0 ............................... [lun0 block/my_iscsi_pv (rw)]

Next, we create the ACLs for the target. Notice, I used the ACL of the iSCSI client we set up earlier. This needs to be repeated for all clients that need access to the target. In OpenShift, this means ALL nodes that could have a PersistentVolumeClaim against the PV. So, if you have 12 nodes, then there needs to be 12 client IQNs added to the ACL.

/iscsi/iqn.20...ipv/tpg1/acls> cd /
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- my_iscsi_pv ....................................................... [/dev/mapper/my_vg-my_lv (1.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2017-04.com.example.mydomain:myiscsipv ........................................................................ [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1994-05.com.redhat:d1fa8aab755 ................................................................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ........................................................................... [lun0 block/my_iscsi_pv (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 .................................................................... [block/my_iscsi_pv (/dev/mapper/my_vg-my_lv)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json

Now we’ve completed the basic setup, and the above shows our completed tree. This configuration is now active, and we can move on to testing the target.

Before moving on, you’ll notice we did not modify the portal part of the TPG. The default 0.0.0.0:3260 means that this target is accessible from any IP on the iSCSI server via port 3260. If desired, we could change the port or restrict the listening IP addresses, but for a “quick-and-dirty” setup, this is unnecessary.

Testing a LUN

To test the target, let’s log into our iSCSI client, and see if we can see see the volume. To do this, we’ll use the iscsiadm command.

First, check the initiator name to make sure it matches what you have in the ACL for the TPG. If it does not match, then this test will not work.

[root@iscsi-client ~]# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:d1fa8aab755

Next, use the iscsiadm command to discover all targets on the iSCSI server.

[root@iscsi-client ~]#  iscsiadm --mode discoverydb --type sendtargets --portal iscsi-server.example.com --discover
172.31.0.127:3260,1 iqn.2017-04.com.example.mydomain:myiscsipv

Next, we login to the target.

[root@iscsi-client ~]# iscsiadm --mode node --targetname iqn.2017-04.com.example.mydomain:myiscsipv  --portal iscsi-server.example.com:3260 --login
Logging in to [iface: default, target: iqn.2017-04.com.example.mydomain:myiscsipv, portal: 172.31.0.127,3260] (multiple)
Login to [iface: default, target: iqn.2017-04.com.example.mydomain:myiscsipv, portal: 172.31.0.127,3260] successful.

If the login command is successful, we should see a new block device attached to the host. In /var/log/messages, there should be messages similar to the ones below:

Apr 18 23:53:31 iscsi-client kernel: scsi host2: iSCSI Initiator over TCP/IP
Apr 18 23:53:31 iscsi-client kernel: scsi 2:0:0:0: Direct-Access     LIO-ORG  my_iscsi_pv      4.0  PQ: 0 ANSI: 5
Apr 18 23:53:31 iscsi-client kernel: scsi 2:0:0:0: alua: supports implicit and explicit TPGS
Apr 18 23:53:31 iscsi-client kernel: scsi 2:0:0:0: alua: port group 00 rel port 01
Apr 18 23:53:31 iscsi-client kernel: scsi 2:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA
Apr 18 23:53:31 iscsi-client kernel: scsi 2:0:0:0: alua: Attached
Apr 18 23:53:31 iscsi-client kernel: scsi 2:0:0:0: Attached scsi generic sg0 type 0
Apr 18 23:53:31 iscsi-client kernel: sd 2:0:0:0: [sda] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
Apr 18 23:53:31 iscsi-client kernel: sd 2:0:0:0: [sda] Write Protect is off
Apr 18 23:53:31 iscsi-client kernel: sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Apr 18 23:53:31 iscsi-client kernel: sd 2:0:0:0: [sda] Attached SCSI disk
Apr 18 23:53:31 iscsi-client iscsid: Could not set session1 priority. READ/WRITE throughout and latency could be affected.
Apr 18 23:53:31 iscsi-client iscsid: Connection1:0 to [target: iqn.2017-04.com.example.mydomain:myiscsipv, portal: 172.31.0.127,3260] through [iface: default] is operational now

In the logs, we see this device is assigned to sda, so we should see sda in the output of lsblk.

[root@iscsi-client ~]# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda      8:0    0   1G  0 disk 
vda    253:0    0  50G  0 disk 
└─vda1 253:1    0  50G  0 part 
vdb    253:16   0  50G  0 disk 
└─vdb1 253:17   0  50G  0 part /

This connection will persist through reboot. Since this is just a test, we don’t want that. To break the connection, we’ll logout of the target using iscsiadm.

[root@iscsi-client ~]# iscsiadm --mode node --targetname iqn.2017-04.com.example.mydomain:myiscsipv  --portal iscsi-server.example.com:3260 --logout
Logging out of session [sid: 1, target: iqn.2017-04.com.example.mydomain:myiscsipv, portal: 172.31.0.127,3260]
Logout of [sid: 1, target: iqn.2017-04.com.example.mydomain:myiscsipv, portal: 172.31.0.127,3260] successful.

After logging out of the target, we should no longer see sda attached to our system.

[root@iscsi-client ~]# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  50G  0 disk 
└─vda1 253:1    0  50G  0 part /
vdb    253:16   0  50G  0 disk 
└─vdb1 253:17   0  50G  0 part 

OpenShift PersistentVolume Configuration

Now that we’ve tested our iSCSI target, we can create an OpenShift PersistentVolume. Compared to the previous steps, this is a piece of cake! Below is example YAML, based upon our previous work:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: iscsi-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  iscsi:
     targetPortal: iscsi-server.example.com
     iqn: iqn.2017-04.com.example.mydomain:myiscsipv
     lun: 0
     fsType: 'ext4'
     readOnly: false