ZFS
ZFS setting on clondaq2 (RHEL9)
Install zfs packages:
yum install zfs
Get the list of available disks:
fdisk -l
Load zfs modules:
/sbin/modprobe zfs
Create raidz-2 storage pool:
zpool create data raidz2 nvme2n1 nvme3n1 nvme4n1 nvme5n1 nvme6n1 nvme7n1 nvme8n1 nvme9n1 nvme10n1 nvme11n1
Check it by running command zfs list, you should see following:
NAME USED AVAIL REFER MOUNTPOINT data 823K 106T 219K /data
NOTES: Useful command: zpool status. Run zpool scrub on a regular basis to identify data integrity problems.
old Solaris settings
RAID partitions on clondaq2
Four zfs partitions made on clonfs raid system connected to clondaq2: raid0, raid1, raid2 and raid3.
NOTE: at some point (after machine was renamed from clonxt3 and several reboots) raid3 dissapeared. When I tried to created it said it exist, but mount did not worked. Maybe it happened when I was playing with the same partition from clondaq1. Anyway, I created raid3 again, will watch it ...
To create raid0-raid3 on clondaq2:
zpool create -f raid0 c0t600C0FF00000000009839205A879C400d0s2 zpool create -f raid1 c0t600C0FF0000000000983926ED5760700d0s2
zpool create -f raid2 c0t600C0FF000000000098392374595CB00d0s2 zpool create -f raid3 c0t600C0FF00000000009839252F7ADDC00d0s2
#zpool create -f raid2 c0t600C0FF00000000009839260D9481700d0s2 #zpool create -f raid3 c0t600C0FF0000000000983927F9F306900d0s2
#zpool create -f raid2 c0t600C0FF00000000009839223AFDC0A00d0s2 #zpool create -f raid3 c0t600C0FF0000000000983920939EB6300d0s2
zfs set mountpoint=/mnt/raid0 raid0 zfs set mountpoint=/mnt/raid1 raid1 zfs set mountpoint=/mnt/raid2 raid2 zfs set mountpoint=/mnt/raid3 raid3
To destroy for example raid3 on clondaq2:
zfs umount /mnt/raid3 zpool destroy raid3
To display the list of all zfs partitions:
zfs list
NOTE: After reboot zfs partitions maybe (will be ?) unmounted, for example on clon10 'zfs list' returns following:
NAME USED AVAIL REFER MOUNTPOINT data 134G 214G 134G /data data1 131K 182G 49K /data1
but /data and /data1 do not exists or empty. To mount those two partitions, do following as 'root':
zfs mount data zfs mount data1
RAIDOLD on clon10
mkdir /raidold zpool create -f raidold \ c1t32d0 c1t33d0 c1t34d0 c1t35d0 c1t36d0 c1t37d0 c1t38d0 c1t39d0 c1t40d0 c1t41d0 c1t42d0 \ c1t48d0 c1t49d0 c1t50d0 c1t51d0 c1t52d0 c1t53d0 c1t54d0 c1t55d0 c1t56d0 c1t57d0 c1t58d0 \ c1t64d0 c1t65d0 c1t66d0 c1t67d0 c1t68d0 c1t69d0 c1t70d0 c1t71d0 c1t72d0 c1t73d0 c1t74d0 \ c1t80d0 c1t81d0 c1t82d0 c1t83d0 c1t84d0 c1t85d0 c1t86d0 c1t87d0 c1t88d0 c1t89d0 c1t90d0 zfs set mountpoint=/raidold raidold
extra disks on clons
clon01:
zpool create -f space c1t1d0 zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 17.9G 49.0G 94K /rpool rpool/ROOT 8.94G 49.0G 18K legacy rpool/ROOT/s10s_u6wos_07b 8.94G 49.0G 6.42G / rpool/ROOT/s10s_u6wos_07b/var 2.53G 49.0G 2.53G /var rpool/dump 1.00G 49.0G 1.00G - rpool/export 38K 49.0G 20K /export rpool/export/home 18K 49.0G 18K /export/home rpool/swap 8G 57.0G 24K - space 89.5K 66.9G 1K /space
To change swap size for example, do following:
zfs set volsize=4G rpool/swap
To add new disk to the existing zfs pool, do following:
iostat -E (shows all disks available) zpool add rpool ...
Hardware Maintenance
Administrator's Manual: pdf (hardware map see page 94)
To check disks status, run:
zpool status
To offline a failing drive, run:
zpool offline pool-name disk-name
(A -t flag allows the disk to come back online after a reboot.)
Unconfigure disk. For example if disk name is 'c0t1d0', run:
cfgadm -c unconfigure c0::dsk/c0t1d0
On sfs61: Open a cover and look for the disk with blue led on. Replace it with new disk. Do not keep cover opened more then 90 sec !.
Once the drive has been physically replaced, run the replace command against the device:
zpool replace pool-name device-name
(for example 'zpool replace export /dev/dsk/c0t3d0'; it will take a while, 'zpool status' shows that disk 'resilvered').
After an offlined drive has been replaced, it can be brought back online:
zpool online pool-name disk-name
It is good idea to run:
zpool scrub pool-name
Firmware upgrades may cause the disk device ID to change. ZFS should be able to update the device ID automatically, assuming that the disk was not physically moved during the update. If necessary, the pool can be exported and re-imported to update the device IDs.
from the web
Useful link: http://www.princeton.edu/~unix/Solaris/troubleshoot/zfs.html
Ok, ZFS is now in the tree, what's now? Below you'll find some
instructions how to quickly make it up and running.
First of all you need some disks. Let's assume you have three spare SCSI disks: da0, da1, da2.
Add a line to your /etc/rc.conf to start ZFS automatically on boot:
# echo 'zfs_enable="YES"' >> /etc/rc.conf
Load ZFS kernel module, for the first time by hand:
# kldload zfs.ko
Now, setup one pool using RAIDZ:
# zpool create tank raidz da0 da1 da2
It should automatically mount /tank/ for you.
Ok, now put /usr/ on ZFS and propose some file systems layout. I know you probably have some files already, so we will work on /tank/usr directory and once we ready, we will just change the mountpoint to /usr.
# zfs create tank/usr
Create ports/ file system and enable gzip compression on it, because most likely we will have only text files there. On the other hand, we don't want to compress ports/distfiles/, because we keep compressed stuff already in-there:
# zfs create tank/usr/ports # zfs set compression=gzip tank/usr/ports # zfs create tank/usr/ports/distfiles # zfs set compression=off tank/usr/ports/distfiles
(You do see how your life is changing, don't you?:))
Let's create home file system, my own home/pjd/ file system. I know we use RAIDZ, but I want to have directory where I put extremly important stuff, you I'll define that each block has to be stored in tree copies:
# zfs create tank/usr/home # zfs create tank/usr/home/pjd # zfs create tank/usr/home/pjd/important # zfs set copies=3 tank/usr/home/pjd/important
I'd like to have directory with music, etc. that I NFS share. I don't really care about this stuff and my computer is not very fast, so I'll just turn off checksumming (this is only for example purposes! please, benchmark before doing it, because it's most likely not worth it!):
# zfs create tank/music # zfs set checksum=off tank/music # zfs set sharenfs=on tank/music
Oh, I almost forget. Who cares about access time updates?
# zfs set atime=off tank
Yes, we set it only on tank and it will be automatically inherited by others.
Will be also good to be informed if everything is fine with our pool:
# echo 'daily_status_zfs_enable="YES"' >> /etc/periodic.conf
For some reason you still need UFS file system, for example you use ACLs or extended attributes which are not yet supported by our ZFS. If so, why not just use ZFS to provide storage? This way we gain cheap UFS snapshots, UFS clones, etc. by simply using ZVOLs.
# zfs create -V 10g tank/ufs # newfs /dev/zvol/tank/ufs # mount /dev/zvol/tank/ufs /ufs
# zfs snapshot tank/ufs at 20070406 # mount -r /dev/zvol/tank/ufs at 20070406 /ufs20070406
# zfs clone tank/ufs at 20070406 tank/ufsok # fsck_ffs -p /dev/zvol/tank/ufsok # mount /dev/zvol/tank/ufsok /ufsok
Want to encrypt your swap and still use ZFS? Nothing more trivial:
# zfs create -V 4g tank/swap # geli onetime -s 4096 /dev/zvol/tank/swap # swapon /dev/zvol/tank/swap.eli
Trying to do something risky with your home? Snapshot it first!
# zfs snapshot tank/home/pjd at justincase
Turns out it was more stupid than risky? Rollback your snapshot!
# zfs rollback tank/home/pjd at justincase # zfs destroy tank/home/pjd at justincase
Ok, everything works, we may set tank/usr as our real /usr:
# zfs set mountpoint=/usr tank/usr
Don't forget to read zfs(8) and zpool(8) manual pages and SUN's ZFS administration guide:
http://www.opensolaris.org/os/community/zfs/docs/zfsadmin.pdf
--
================================== from the web =======================================
ZFS and Containers: An Example This section is a step-by-step guide that shows how to perform certain ZFS file system tasks inside of Solaris Containers; for example, taking snapshots and managing data compression. It does this by going through the following steps:
* Creating a zpool * Creating a Zone * Allocating a ZFS File System to a Zone * Creating New File Systems * Applying Quota to the File Systems * Changing the Mountpoint of a File System * Setting the Compression Property * Taking a Snapshot
Each of these steps is described in detail below.
Creating a zpool ZFS uses device names or partition names when dealing with pools and devices. For a device, this will be something like c1t0d0 (for a SCSI device) or c1d1 (for an IDE device). For a partition, it will be something like c1t0d0s0 (for a SCSI device) or c1d1s0 (for an IDE device). This example creates a pool that is mirrored using two disks.
1. To create a zpool in the global zone, use the zpool create command. Typically, you use two devices to provide redundancy.
Global# zpool create mypool mirror c2t5d0 c2t6d0
Note that the zpool create command may fail if the devices are in use or contain some types of existing data (e.g. UFS file system). If they are in use, you will need to unmount them or otherwise stop using them. If they contain existing data you can use the -f(force) flag to override the safety check, but be sure that you are not destroying any data you want to retain. 2. Examine the pool properties using the zpool list command.
Global# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool
199G
164K
199G
0%
online
--
This shows you that there is one zpool, named mypool,with a capacity of 199GBytes.
Creating a Zone To show ZFS working in an environment that is isolated from other applications on the system, you need to create a zone. To create a zone:
1. Create a directory where the zone file system will reside, using the mkdir command. Be sure to chose a location where the filesystem has at least 80MBytes of available space.
Global# mkdir /zones
Note that in this example, for the sake of brevity, the root file system of the zone is a UFS file system.
2. Configure the zone (myzone), using the zonecfg command, and specify the location of the zone's files (/zones/myzone). Use the following series of commands.
Global# zonecfg -z myzone
myzone: No such zone configured
Use 'create' to begin configuring a new zone
zonecfg:myzone< create
zonecfg:myzone< set zonepath=/zones/myzone
zonecfg:myzone< verify
zonecfg:myzone< commit
zonecfg:myzone< exit
Again, for the purpose of streamlining, this example uses a very minimal zone. For more details on creating zones see the Solaris Containers How To Do Stuff guide at: sun.com/software/solaris/howtoguides/containersLowRes.jsp
3. Install the zone by using the zoneadm.
Global# zoneadm -z myzone install
Preparing to install zone >myzone<
[Output from zoneadm, this may take a few mins]
4. Boot the zone to complete the installation, using the zoneadm command.
Global# zoneadm -z myzone boot
5. Use the zlogin command to connect to the zone console.
Global# zlogin -C myzone
[Connected to zone 'myzone' console]
[Initial zone boot output, service descriptions are loaded etc.]
It may take a short while for the first boot to configure everything, load all the service descriptors, and so on. You will need to answer the system configuration details. Some suggestions are:
Terminal=(12)X Terminal Emulator (xterms)
Not Networked
No Kerberos
Name service = None
Time Zone = your-time-zone
root passwd = (Your choice—remember it though!)
The zone will reboot after you have provided the configuration information.
6. Before you can proceed to the next stage, the configured zone needs to be shutdown (configuration changes are only applied when the zone boots).
Global# zlogin myzone init 5
Allocating a ZFS File System to a Zone Now that you have a zpool (mypool) and a zone (myzone) you are ready to allocate a ZFS file system to the zone.
1. To create a ZFS file system, use the zfs create command.
Global# zfs create mypool/myzonefs
2. To apply a quota to the file system, use the zfs set quota command.
Global# zfs set quota=5G mypool/myzonefs
The file system and all of its child file systems can be no larger than the designated quota. Note that both these steps must be performed in the global zone. Also notice that creating the file system in ZFS is much simpler than with a traditional file system/volume manager combination.
To illustrate the isolation/security properties of containers with ZFS this example now creates a ZFS file system that will remain outside the container. There is no need to apply a quota to this outside file system.
3. To create this other file system, again use the zfs create command.
Global# zfs create mypool/myfs
4. To show the properties of all the pool and the file systems, use the zfs list command.
Global# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool
396G
197G
99.5K
/mypool
mypool/myfs
98.5K
197GK
98.5K
/mypool/myfs
mypool/myzonefs
98.5K
5G
98.5K
/mypool/myzonefs
To make the file system (myzonefs) available in the zone (myzone), the zone configuration needs to be updated.
5. To update the zone configuration, use the zonecfg command.
Global# zonecfg -z myzone
zonecfg:myzone> add dataset
zonecfg:myzone:dataset> set name=mypool/myzonefs
zonecfg:myzone:dataset> end
zonecfg:myzone> commit
zonecfg:myzone> exit
The mypool/myzonefs file system is now added to the zone configuration. Note that these steps must be performed with the zone shut down, otherwise the zone configuration changes would not be visible until the next reboot. To check that the zone is shut down try logging into it using zlogin myzone. If the zone is shut down the login will fail; if the zone is running you will see a login prompt—login as root and shut the zone down with init 5. These steps are performed in the global zone.
6. Now boot the zone.
Global# zoneadm -z myzone boot
7. Log in to the zone. (Leave a few seconds for the zone to boot.)
Global# zlogin -C myzone
[Connected to zone 'myzone' pts/3]
[Usual Solaris login sequence]
8. List the ZFS file systems in the zone.
NAME USED AVAIL REFER MOUNTPOINT
mypool
0M
200B
--
/mypool
mypool/myzonefs
8K
5G
8K
/mypool/myzonefs
Note the 5GByte maximum available from the external quota and that the other file systems in the pool (mypool/myfs) are not visible. This demonstrates the isolation property that Containers provide.
Creating New File Systems Administering ZFS file systems from the non-global zone is done just like it is in the global zone, although you are limited to operating within the file system that is allocated to the zone (mypool/myzonefs). New ZFS file systems are always created as a child of this file system because this is the only ZFS file system the non-global zone can see. The administrator in the non-global zone can create these file systems. There is no need to involve administrator of the global zone, though the administer could do so if it were necessary.
1. To create a newfile system, use the zfs create command.
MyZone# zfs create mypool/myzonefs/tim
MyZone# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool
594M
197G
99K
/mypool
mypool/myzonefs
197K
5.00G
98.5K
/mypool/myzonefs
mypool/myzonefs/tim
98.5K
5.00G
98.5K
/mypool/myzonefs/tim
The non-global zone administrator can create as many child file systems as s/he wants and each child file system can have its own file systems, and in that way form a hierarchy.
As a demonstration that the non-global zone administrator is limited to the assigned file systems, this example demonstrates trying to break security by creating a file system outside the container's "space".
2. Try to create another file system outside of mypool/myzonefs, using the zfs create command.
MyZone# zfs create mypool/myzonefs1
cannot create 'mypool/myzonefs1': permission denied
As you can see, ZFS and zones security denies permission for the non-global zone to access resources it has not been allocated and the operation fails.
Applying Quota to the File Systems Typically, to prevent the user consuming all of the space, a non-global zone administrator will want to apply a quota to the new file system. Of course, the child's quota can't be more than 5GByte as that's the quota specified by the global zone administrator to all of the file systems below mypool/myzonefs.
1. To set a quota on our new file system, use the zfs set quota command.
MyZone# zfs set quota=1G mypool/myzonefs/tim
MyZone# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool
508M
197G
99K
/mypool
mypool/myzonefs
198K
5.00G
99K
/mypool/myzonefs
mypool/myzonefs/tim
98.5K
1024M
98.5K
/mypool/myzonefs/tim
The administrator of the non-global zone has set the quota of the child file system to be 1G. They have full authority to do this because they are operating on their delegated resources and do not need to involve the global zone administrator.
The ZFS property inheritance mechanism applies across zone boundaries, so the non-global zone administrator can specify his/her own property values should s/he wish to do so. As with normal ZFS property inheritance, these override inherited values.
Changing the Mountpoint of a File System Now that the file system is set up and has the correct quota assigned to it, it is ready for use. However, the place where the file system appears (the mountpoint) is partially dictated by what the global zone administrator initially chose as the pool name (in this example, mypool/myzonefs). But typically, a non-global zone administrator would want to change it.
1. To change the mountpoint, use the zfs set mountpoint command.
MyZone# zfs set mountpoint=/export/home/tim mypool/myzonefs/tim
MyZone# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool
508M
197G
99K
/mypool
mypool/myzonefs
198K
5.00G
99K
/mypool/myzonefs
mypool/myzonefs/tim
98.5K
1024M
98.5K
/export/home/tim
Note that the mountpoint can be changed for any file system independently.
Setting the Compression Property The next example demonstrates the compression property. If compression is enabled, ZFS will transparently compress all of the data before it is written to disk.
The benefits of compression are both saved disk space and possible write speed improvements.
1. To see what the current compression setting is, use the zfs get command.
MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim
NAME PROPERTY VALUE SOURCE
mypool
compression
off
default
mypool/myzonefs
compression
off
default
mypool/myzonefs/tim
compression
off
default
Beaware that the compression property on the pool is inherited by the file system and child file system. So if the non- global zone administrator sets the compression property for the delegated file system, it will set it for everything below, as well.
2. To set the compression for the file system, use the zfs set command.
MyZone# zfs set compression=on mypool/myzonefs
3. Examine the compression property again in the non-global zone.
MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim
NAME PROPERTY VALUE SOURCE
mypool
compression
off
default
mypool/myzonefs
compression
on
local
mypool/myzonefs/tim
compression
off
Inherited from mypool/myzonefs
Note the compression property has been inherited by mypool/myzonefs/tim as with normal ZFS administration.
Taking a Snapshot One of the major advantages of ZFS is the ability to create an instant snapshot of any file system. By delegating a file system to a non-global zone this feature becomes available as an option for the non-global zone administrator.
1. To take a snapshot named "1st" of the file system, use the zfs snapshot command.
MyZone# zfs snapshot mypool/myzonefs@1st
MyZone# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool
512K
99K
default
/mypool
mypool/myzonefs
198K
5.00G
99K
/mypool/myzonefs
mypool/myzonefs@1st
OK
--
99K
--
mypool/myzonefs/tim
98.5K
1024M
98.5K
/export/home/tim
As with ZFS file systems in the global zone, this snapshot is now accessible from the root of the file system in .zfs/snapshot/1st.
Back To Top
Summary Once a zone has been created and a ZFS file system has been allocated to it, the administrator for that (non-global) zone can create file systems, take snapshots, create clones, and perform all the other functions of an administrator—within that zone. Yet the global zone, and any other zones, are fully isolated from whatever happens in that zone.
The integration of Solaris Containers and Solaris ZFS is just another way that the Solaris 10 OS is providing cost benefits to customers by allowing them to safely consolidate applications and more easily manage the data those applications use.