QTS Hero... ZFS? What? When?

Interested in our products? Post your questions here. Let us answer before you buy.
Post Reply
occamsrazor
Know my way around
Posts: 234
Joined: Tue Mar 30, 2010 8:30 pm

QTS Hero... ZFS? What? When?

Post by occamsrazor » Fri Oct 18, 2019 4:40 am

Wasn't really sure where to post this.... but seeing some references online to "QTS Hero" - a ZFS-based version of the operating system??

1. "QTS hero (a ZFS-based NAS operating system) features data inline deduplication and compression, end-to-end data integrity, greater reliability, and higher performance, making it especially useful for tackling video-editing challenges."
https://nascompares.com/2019/09/09/qnap ... announced/

2. At 00:50 they talk about why QNAP hasn't gone for BTRFS, and then go on to mention "plans" for ZFS coming to desktop models.
https://www.youtube.com/watch?v=GGdEO8lNHXQ

2. An image at the bottom of this Hong Kong forum thread refers to models that? are QTS Hero compatible? Or what?
https://www.hkepc.com/forum/viewthread. ... 892&page=1

3. This video in Chinese from QNAP is all about it, but I've no idea what they are saying
https://www.youtube.com/watch?v=Y5LWX8j87qk

I know ZFS functionality exists in some enterprise rackmount models (https://enterprise-nas.qnap.com/en/), but if they were bringing it to some desktop models that would be really interesting...

EDIT: A brief video in English, but with not a lot of information, posted just today: https://live.qnap.com/en/video/show/4674
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

User avatar
dolbyman
Guru
Posts: 20397
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: QTS Hero... ZFS? What? When?

Post by dolbyman » Fri Oct 18, 2019 4:48 am

judging by the quality control of their current firmwares .. it will be several years until they have ZFS running in a stable way

so sit back relax .. and wait for disaster to strike

User avatar
Trexx
Ask me anything
Posts: 5304
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: QTS Hero... ZFS? What? When?

Post by Trexx » Fri Oct 18, 2019 4:50 am

I have an educated hunch that we will start hearing more about QTS Hero in the not too distant future.

Because ZFS is a COMPLETELY different file system/structure than current ext4, having good backups may become very important.


Sent from my iPad using Tapatalk
Paul

Model: TS-877-1600 FW: 4.4.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x 500GB Evo 860
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
GPU: EVGA GTX 1060 6GB
UPS: CP AVR1350

Model:TVS-673 32GB FW: 4.4.3.x Test/Backup Box
Model:TS-228a FW: 4.4.3.x Test/Backup Box
-----------------------------------------------------------------------------------------------------------------------------------------
NAS RAID Rebuild Times | Live QTS Videos | | QNAP NAS Guide | Information needed when you ask for HELP | QNAP Links, Tutorials, etc.
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq

occamsrazor
Know my way around
Posts: 234
Joined: Tue Mar 30, 2010 8:30 pm

Re: QTS Hero... ZFS? What? When?

Post by occamsrazor » Fri Oct 18, 2019 4:51 am

dolbyman wrote:
Fri Oct 18, 2019 4:48 am
judging by the quality control of their current firmwares .. it will be several years until they have ZFS running in a stable way
so sit back relax .. and wait for disaster to strike
Disaster can strike anytime... let it strike on a new shiny ZFS-based OS :-)
But seriously... I find this very exciting... if... it's going to be something that can be used with current models.
PS - See video in English I subsequently added to end of the first post....
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

User avatar
dolbyman
Guru
Posts: 20397
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: QTS Hero... ZFS? What? When?

Post by dolbyman » Fri Oct 18, 2019 4:55 am

occamsrazor wrote:
Fri Oct 18, 2019 4:40 am
EDIT: A brief video in English, but with not a lot of information, posted just today: https://live.qnap.com/en/video/show/4674
If they only had a camera with higher dynamic range or a videographer with a bit more know-how .. you could actually read the slides :lol:
lolqnap.png
You do not have the required permissions to view the files attached to this post.

occamsrazor
Know my way around
Posts: 234
Joined: Tue Mar 30, 2010 8:30 pm

Re: QTS Hero... ZFS? What? When?

Post by occamsrazor » Fri Oct 18, 2019 4:57 am

dolbyman wrote:
Fri Oct 18, 2019 4:55 am
If they only had a camera with higher dynamic range or a videographer with a bit more know-how .. you could actually read the slides :lol:
Was thinking that myself. QNAP is really weird the way they post videos, especially very long ones like an hour long sometimes, before posting any readable information....
Trexx wrote:
Fri Oct 18, 2019 4:50 am
I have an educated hunch that we will start hearing more about QTS Hero in the not too distant future.
Because ZFS is a COMPLETELY different file system/structure than current ext4, having good backups may become very important. Image
There's a lot of "Ifs" for sure, but ZFS has some pretty cool features and if QNAP could bring it to mainstream NAS models that would be a significant advantage.
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

occamsrazor
Know my way around
Posts: 234
Joined: Tue Mar 30, 2010 8:30 pm

Re: QTS Hero... ZFS? What? When?

Post by occamsrazor » Fri Oct 18, 2019 5:26 am

This Twitter user has posted a bunch of slides in French: https://twitter.com/inpacthardware/stat ... 7309058048

On one of the later slides it shows "Other models that accept QTS Hero"
https://pbs.twimg.com/media/EGhEVFxXoAA ... =4096x4096
A fair range... they say all recent AMD/Intel models.
My crappy translation of the notes below... "4GB of RAM is needed and SSD recommended, 8GB of RAM is necessary for deduplication - 32GB recommended"
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 9389
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: QTS Hero... ZFS? What? When?

Post by Moogle Stiltzkin » Fri Oct 18, 2019 7:16 am

the hero we deserve... :D

well the slide show in that video claims 2019, and now is already october, so somewhere between this month and December? which isn't long.

one thing i like about zfs is the end to end checksum verification. and it can self repair. and also....report on file corruption rather than stay silent about issues.

i wonder though if these new models will support ecc ram.
ECC RAM or Not?
This is probably the most contested issue surrounding ZFS (the filesystem that FreeNAS uses to store your data) today. I’ve run ZFS with ECC RAM and I’ve run it without. I’ve been involved in the FreeNAS community for many years and have seen people argue that ECC is required and others argue that it is a pointless waste of money. ZFS does something no other filesystem you’ll have available to you does: it checksums your data, and it checksums the metadata used by ZFS, and it checksums the checksums. If your data is corrupted in memory before it is written, ZFS will happily write (and checksum) the corrupted data. Additionally, ZFS has no pre-mount consistency checker or tool that can repair filesystem damage. This is very nice when dealing with large storage arrays as a 64TB pool can be mounted in seconds, even after a bad shutdown. However if a non-ECC memory module goes haywire, it can cause irreparable damage to your ZFS pool that can cause complete loss of the storage. For this reason, I highly recommend the use of ECC RAM with “mission-critical” ZFS. Systems with ECC RAM will correct single bit errors on the fly, and will halt the system before they can do any damage to the array if multiple bit errors are detected. If it’s imperative that your ZFS based system must always be available, ECC RAM is a requirement. If it’s only some level of annoying (slightly, moderately…) that you need to restore your ZFS system from backups, non-ECC RAM will fit the bill.
https://www.ixsystems.com/blog/a-comple ... practices/


perhaps it will be ok with backup i suppose for the older models that don't support ecc ram? :'


and how much ram is needed
How Much RAM is needed?
FreeNAS requires 8 GB of RAM for the base configuration. If you are using plugins and/or jails, 12 GB is a better starting point. There’s a lot of advice about how RAM hungry ZFS is, how it requires massive amounts of RAM, an oft quoted number is 1GB RAM per TB of storage. The reality is, it’s complicated. ZFS does require a base level of RAM to be stable, and the amount of RAM it needs to be stable does grow with the size of the storage. 8GB of RAM will get you through the 24TB range. Beyond that 16GB is a safer minimum, and once you get past 100TB of storage, 32GB is recommended. However, that’s just to satisfy the stability side of things. ZFS performance lives and dies by its caching. There are no good guidelines for how much cache a given storage size with a given number of simultaneous users will need. You can have a 2TB array with 3 users that needs 1GB of cache, and a 500TB array with 50 users that need 8GB of cache. Neither of those scenarios are likely, but they are possible. The optimal cache size for an array tends to increase with the size of the array, but outside of that guidance, the only thing we can recommend is to measure and observe as you go. FreeNAS includes tools in the GUI and the command line to see cache utilization. If your cache hit ratio is below 90%, you will see performance improvements by adding cache to the system in the form of RAM or SSD L2ARC (dedicated read cache devices in the pool).

good thing i got 16gb ram :mrgreen: deduplication seems like a kewl feature, but i'm not sure i have enuff ram :'
ZFS Dedupe: The Cost
Saving space through deduplication doesn't come for free. There is a cost. In the case of ZFS, it's memory: ZFS keeps a dedup table in which it stores all the checksums of all the blocks that were written after deduplication was enabled. When writing new blocks, it uses this table to determine whether a block has been written yet, or not.

Over time, the table becomes larger and larger, and since every write operation has to use it, it should be kept in main memory to avoid unnecessary extra reads from disk. To be clear: ZFS can work perfectly well even if the table is not in memory. But the bigger the deduplication table grows, the slower write performance will become, as more and more writes trigger more and more extra reads for dedup table lookups.

How much memory does one need to keep the ZFS dedup table in memory, and hence your system happy?

According to the ZFS dedup FAQ, each entry in the dedup table costs about 320 Bytes of memory per block. To estimate the size of the dedup table, we need to know how many blocks ZFS will need to store our data. This question can be tricky: ZFS uses a variable block size between 512 bytes and 128K, depending on the size of the files it stores. So we can't really know in advance how many blocks ZFS will use for storing our data.

If we mainly store large files (think videos, photos, etc.), then the average block size will be closer to 128K, if we store small files (source code, emails, other data), we'll probably be closer to a few K. Here are some ways to find out for sure:
https://constantin.glez.de/2011/07/27/z ... ot-dedupe/


so is this zfs linux? or is it gonna be Bsd? :'
https://www.qnap.com/qes/2.1.0/en/

5. ZFS Support
One BSD user noted that one of the main reasons that he switched to BSD was ZFS. In fact almost all of the people I talked to mentioned ZFS support on BSD as the reason they did not return to Linux.

This is an area where Linux will lose out on for the time being. While OpenZFS is available on some Linux distros, ZFS is built into the BSD kernels. This alone means that ZFS will have better performance on BSD. While there have been several attempts to get ZFS into the Linux kernel, licensing issues will be solved first.

6. License
There was also a difference of opinion on licenses. The general idea held by many is the GPL is not truly free because it put limits on how you can make use of the software. Some also think that the GPL is “too large and difficult to interpret which can lead to legal problems down the road if a person is not careful when developing a product with this license”.

On the other hand, the BSD license only has three clauses and allows anyone to “take the software, make changes, and do whatever you want with it, but it also offers protection to the developer”.
https://itsfoss.com/why-use-bsd/
7. Performance
Performance is one of the most crucial factors when it comes to choosing between FreeBSD vs Linux as your primary system. Thankfully, both the systems are known to perform exceptionally well in various performance measurement metrics.

Although very powerful in their own right, both systems perform individually well than the other in some particular areas. FreeBSD has very lower latency regarding networking protocols. This is the reason streaming behemoth Netflix sided with FreeBSD as their primary server component.

Linux, on the other hand, delivers extra speed and performance when running native applications. As Linux is much more application-centric than FreeBSD ever will be and has co-operative hardware support from corporations like IBM and Intel, Linux systems always tend to run their applications slightly faster than their BSD counterparts.
https://www.ubuntupit.com/freebsd-vs-li ... he-system/



is it true that with zfs once you set your raid or should i say zraid (raid5) / zraid2(raid6)? to use a certain number of hdds, you cannot expand from that without trashing your raid and starting from scratch? well i can live with that (backups :) ). but just checking....



based on that video, they showed the qts hero options for that zfs. i guess they are like enable deduplication and stuff. not sure what other options is there, never used zfs before :' i wonder if there is an option to point you to any problematic files due to corruption?
Last edited by Moogle Stiltzkin on Fri Oct 18, 2019 10:41 am, edited 1 time in total.
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

Bob Zelin
Been there, done that
Posts: 930
Joined: Mon Nov 21, 2016 12:55 am
Location: Orlando, FL.
Contact:

Re: QTS Hero... ZFS? What? When?

Post by Bob Zelin » Fri Oct 18, 2019 8:34 am

just came from the QNAP Technology 2020 day in San Jose, California. Yes, ZFS looks great, and will be released this December (possible license fee).
You are not going to want to reformat your QNAP from ground zero, so this will be for new enterprise systems with a lot of RAM. IT will support Z1, Z2 and Z3 (3 drives can fail). The user interface will look almost identical to QTS 4.4.1, but some things have changed. It "should " be faster an more efficient, and you will be able to choose between QTS and ZFS during the initial setup. So, it "looks good" - but then, QTS 4.4.1 looked good, and many of us know better now.
As Dolbyman observes - the first release will probably be a total nightmare for those brave enough to do it (which will probably be me, on some poor schmucks new system) - but it's easy to make mistakes, and possibly start all over, if they don't have any data on their system yet !

At least I confirmed they will not use the BTRFS file system, which is a total nightmare.
Bob Zelin
Bob Zelin / Rescue 1, Inc.
http://www.bobzelin.com

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 9389
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: QTS Hero... ZFS? What? When?

Post by Moogle Stiltzkin » Fri Oct 18, 2019 1:31 pm

i spotted this

How do you remove Container Station data in QTS hero?
Release date: 2019-08-28

https://www.qnap.com/en/how-to/faq/arti ... -qts-hero/


could be more of these nuggets mentioning qts hero...
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

occamsrazor
Know my way around
Posts: 234
Joined: Tue Mar 30, 2010 8:30 pm

Re: QTS Hero... ZFS? What? When?

Post by occamsrazor » Fri Oct 18, 2019 1:43 pm

Bob Zelin wrote:
Fri Oct 18, 2019 8:34 am
Yes, ZFS looks great, and will be released this December (possible license fee).
If so, hopefully something reasonable...
Bob Zelin wrote:
Fri Oct 18, 2019 8:34 am
You are not going to want to reformat your QNAP from ground zero, so this will be for new enterprise systems with a lot of RAM.
Do you mean just because of workflow - not compatibility issues? Because the slides indicate QTS Hero will be available for existing models, including desktop models, not only "new enterprise systems".... Personally I'd be fine to re-intialize my NAS as it's containing almost only backups of other drives.

A lot still to be known, for sure. But for example to be able to have all the functions of ZFS such as self-healing on something like TVS-872XT would be very cool.
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

occamsrazor
Know my way around
Posts: 234
Joined: Tue Mar 30, 2010 8:30 pm

Re: QTS Hero... ZFS? What? When?

Post by occamsrazor » Fri Oct 18, 2019 2:05 pm

Moogle Stiltzkin wrote:
Fri Oct 18, 2019 7:16 am
well the slide show in that video claims 2019, and now is already october, so somewhere between this month and December? which isn't long.
One of the French twitter posters wrote end of 2019 / early 2020. Still not that far away.
Moogle Stiltzkin wrote:
Fri Oct 18, 2019 7:16 am
one thing i like about zfs is the end to end checksum verification. and it can self repair. and also....report on file corruption rather than stay silent about issues.
Exactly... ZFS has some great features very few, if any, other file systems can match. Also inline deduplication and compression (though for my type of data I don't think that'd be much use).
Moogle Stiltzkin wrote:
Fri Oct 18, 2019 7:16 am
i wonder though if these new models will support ecc ram.
This slide implies some type of ECC Ram model offered, or ECC Ram compatibility: https://pbs.twimg.com/media/EGg_3PBWwAA ... =4096x4096
ECC RAM or Not?
This is probably the most contested issue surrounding ZFS ......

....perhaps it will be ok with backup i suppose for the older models that don't support ecc ram?
I won't get into this debate because I simply don't know enough. But from what I can see of the ECC/ZFS debate, while there is a theoretical advantage to using ECC, that in the case you don't use ECC Ram you're still not going to be worse off in terms of data integrity than if you were using a non-checksumming non-self-healing filesystem like EXT4 etc. My point being for those on a budget, ZFS with non-ECC RAM is still going to be better (ignoring any bugs) than EXT4 in terms of data integrity... Just from what I have read, I have no actual experience with ZFS.
How Much RAM is needed?
....
good thing i got 16gb ram :mrgreen: deduplication seems like a kewl feature, but i'm not sure i have enuff ram :'
I suspect I'll never be able to afford enough RAM for dedup either. But also it's going to depend a lot on what your data is. If it's mostly unique and compressed files such as movies/music then seems like any savings would be minimal anyway, at a big RAM and CPU cost.
Moogle Stiltzkin wrote:
Fri Oct 18, 2019 7:16 am
so is this zfs linux? or is it gonna be Bsd? :'
https://www.qnap.com/qes/2.1.0/en/
One BSD user noted that one of the main reasons that he switched to BSD was ZFS. In fact almost all of the people I talked to mentioned ZFS support on BSD as the reason they did not return to Linux.
That's a good question. While QES may be FreeBSD based, the fact those slides say Appcenter functionality is maintained under QTS Hero makes me think it has to be Linux:
https://pbs.twimg.com/media/EGg_3PIX0AA ... =4096x4096
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 9389
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: QTS Hero... ZFS? What? When?

Post by Moogle Stiltzkin » Fri Oct 18, 2019 2:56 pm

occamsrazor wrote:
Fri Oct 18, 2019 1:43 pm
Bob Zelin wrote:
Fri Oct 18, 2019 8:34 am
Yes, ZFS looks great, and will be released this December (possible license fee).
If so, hopefully something reasonable...
do you mean like the previous exfat license thing, and the mcaffee anti virus those kinds of things? ..... well i think it's worth it for the zfs :' i just hope it's priced reasonably.

occamsrazor wrote:
Fri Oct 18, 2019 1:43 pm
Bob Zelin wrote:
Fri Oct 18, 2019 8:34 am
You are not going to want to reformat your QNAP from ground zero, so this will be for new enterprise systems with a lot of RAM.
Do you mean just because of workflow - not compatibility issues? Because the slides indicate QTS Hero will be available for existing models, including desktop models, not only "new enterprise systems".... Personally I'd be fine to re-intialize my NAS as it's containing almost only backups of other drives.

A lot still to be known, for sure. But for example to be able to have all the functions of ZFS such as self-healing on something like TVS-872XT would be very cool.
if i were to transition from a qts to this qts hero, i would think a reinitialization would be in order i thought? :' does seem troublesome, but i got backups so i can recover after a day or 2 using HBS restore.




occamsrazor wrote:
Fri Oct 18, 2019 2:05 pm
One of the French twitter posters wrote end of 2019 / early 2020. Still not that far away.
Q1 2020 then :(

occamsrazor wrote:
Fri Oct 18, 2019 2:05 pm
Exactly... ZFS has some great features very few, if any, other file systems can match. Also inline deduplication and compression (though for my type of data I don't think that'd be much use).
i do have some data which is very similar copies with minor edits. so dedupe would definitely help me. but from what i understand the tables are stored in the ram, so i'm wondering what happens in the event of a sudden power loss shut down. will that data stored on the ram get lost or errored? cauz if it's anything like the read/write mode for SSDs where sudden power loss for write is VERY very likely to cause data loss, then i probably shouldn't use dedupe, cause i don't have ups. i'm just assuming :'


occamsrazor wrote:
Fri Oct 18, 2019 2:05 pm
This slide implies some type of ECC Ram model offered, or ECC Ram compatibility: https://pbs.twimg.com/media/EGg_3PBWwAA ... =4096x4096
well thats good. because when using zfs there is bound to be a faction that will insist on using ecc ram.

occamsrazor wrote:
Fri Oct 18, 2019 2:05 pm
I won't get into this debate because I simply don't know enough. But from what I can see of the ECC/ZFS debate, while there is a theoretical advantage to using ECC, that in the case you don't use ECC Ram you're still not going to be worse off in terms of data integrity than if you were using a non-checksumming non-self-healing filesystem like EXT4 etc. My point being for those on a budget, ZFS with non-ECC RAM is still going to be better (ignoring any bugs) than EXT4 in terms of data integrity... Just from what I have read, I have no actual experience with ZFS.
thats my thoughts exactly. it's probably still far better than ext4 .... also as long as you have a backup might be ok ?? :'

occamsrazor wrote:
Fri Oct 18, 2019 2:05 pm
I suspect I'll never be able to afford enough RAM for dedup either. But also it's going to depend a lot on what your data is. If it's mostly unique and compressed files such as movies/music then seems like any savings would be minimal anyway, at a big RAM and CPU cost.
then i wonder if this dedupe feature can be toggled on and off for the user to check the space savings to decide which way to go. or will this entail having to reinitialize each time they change this setting? this is the question i'm wondering about :S

occamsrazor wrote:
Fri Oct 18, 2019 2:05 pm

That's a good question. While QES may be FreeBSD based, the fact those slides say Appcenter functionality is maintained under QTS Hero makes me think it has to be Linux:
https://pbs.twimg.com/media/EGg_3PIX0AA ... =4096x4096
if we can get zfs on linux that would be amazing. but i would be fine with freebsd. although i'm not too familiar with it :' that's why i was reading about it.
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 9389
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: QTS Hero... ZFS? What? When?

Post by Moogle Stiltzkin » Fri Oct 18, 2019 3:15 pm

occamsrazor wrote:
Fri Oct 18, 2019 2:05 pm

That's a good question. While QES may be FreeBSD based, the fact those slides say Appcenter functionality is maintained under QTS Hero makes me think it has to be Linux:
https://pbs.twimg.com/media/EGg_3PIX0AA ... =4096x4096
in that slide you linked they mentioned openzfs, so i was trying to figure out do they mean it's gonna be freebsd or a linux zfs. I initially though freebsd because thats what QES is based on, but now i'm unsure because of what you mentioned.


but between the 2 what is the best option? i was reading some interesting stuff here



FreeBSD moving to ZFS-on-Linux
I don't think HAMMER will come to FreeBSD, and I see no reason that ZFS will go anywhere. It's just a bit of a shock that we're suddenly going to be taking ZFS code targeted specifically at Linux, then trying to undo any Linuxisms to merge into FreeBSD. Apart from being available in base, OpenZFS on FreeBSD will now always be lagging in features and fixes compared to ZoL. I do of course hope that the developers working on ZoL will make a conscious effort to keep any ongoing work as portable as possible, so that the various operating systems can keep up to date with minimum issues.

It's been a day and already there's talk that the current FreeBSD version has the only mature TRIM support, which now needs to be re-implemented in a way that will fit in with the ZoL code.
First, rebasing ZFS on ZoL does not mean stopping ZFS specific development on FreeBSD.

As I understand, ZoF is not a native FreeBSD project (It use OpenZFS), so ZoF will use ZoL as base instead of OpenZFS. Is not that simple ?

Second, it is an opportunity to ask ZoL to be aware about portability and linuxisme. If the ZoL "team" (I don't know how this project is driven) accept the portability challenge, this is a very good thing for FreeBSD, but either OpenBSD, Mac OS X ...

Third, if ZoL is a robust and portable solution, FreeBSD developers can work in other projects.


I discover FreeBSD because I hear Mac OS X use a lot of FreeBSD core and the port manager was, for me, better than the yum installer. After that I discover the update and the documentation ... FreeBSD is not only ZFS and base the ZFS support on FreeBSD from ZoL instead of OpenZFS will not change the status of ZFS in FreeBSD...
I can understand both sides, but I am concerned about how long it will take for ZoF (The acronym I've seen for ZFS On Linux (on FreeBSD)) to be shaken out. Look at all the still-ongoing discussion about ZFS memory management and race conditions on FreeBSD after how many years. Now rip out one version of ZFS, put in another that has been focused on Linux memory management best practices, and wait for the dust to settle...

I really hope it goes well. I certainly understand the motivation, given the momentum that ZoL has been developing, and the additional features they seem to keep adding, to "rebase" off of ZoL. The fact that it sounds like a version of it is up and running is somewhat reassuring, and the indication that Behlendorf (ZoL maintainer) is willing to have the FreeBSD switches (#ifdefs, etc) in the mainline is also a promising sign. FreeBSD also likes to boot off ZFS, and BEs (another major selling point) rely on it. Now the bootcode needs to be touched again (non-trivial; check out the 12.0 upgrade messages on this board) to handle the new ZoF versions, and likely more frequently as they keep adding features.

All the other goodness of FreeBSD still exists. I can still actually read the code and find and change things when I want to. The unified base system rather than Kernel / Distro(user) split. The ability for mortals to build the entire system with the tweaks they would like. The ports tree.

But "we've got the most stable ZFS" (impression or truthiness I leave to you) will likely no longer be the case. And for some, that was all that mattered.
https://forums.freebsd.org/threads/free ... nux.68803/

The New ZFS on FreeBSD Implementation Can Now Be Tested With TrueOS

BSD -- It was recently decided that FreeBSD's ZFS file-system support would be re-based atop ZFS On Linux. That new "ZFS On BSD" implementation based on ZOL continues moving along and it's now easier to test thanks to iX Systems and their TrueOS platform.

With the ZFS On Linux code-base being more actively maintained and improved upon than the OpenZFS support within the Illumos kernel, FreeBSD developers are working on merging their "ZOB" changes with ZOL.

This new ZFS port for FreeBSD will already provide them with vectorized raidz, vectorized checksums, encrypted data-sets, project quotas, and other capabilities not currently offered by their file-system kernel driver.

While this new ZFS code is still very much a work-in-progress, the folks at iX Systems have begun spinning a daily snapshot of their FreeBSD-based TrueOS platform with the new ZFS code.

From zfsonfreebsd.github.io are now daily ISO snapshots of the latest ZoF code paired with TrueOS while disabling the "legacy" ZFS code. I'll add it to my benchmark TODO list soon to give the latest snapshot a whirl.
https://www.phoronix.com/scan.php?page= ... rueOS-Spin



well i just hope they go with the zfs implementation that is stable and has the best updates which they can integrated into this qts HERO. but i'm sure their development team has that already covered and taken into account :'
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 9389
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: QTS Hero... ZFS? What? When?

Post by Moogle Stiltzkin » Fri Oct 18, 2019 5:06 pm

The Future of OpenZFS and FreeBSD

An overview of the forthcoming changes to the OpenZFS Project, and how FreeBSD will interact with the OpenZFS Project.

This talk will discuss:

-How the OpenZFS project has changed
-New problems as ZFS has matured (deprecation policy)
-How the OpenZFS project is working to reduce the differences across platforms (command line switches, NFS interoperability)
-Interoperability improvements (feature flag 'compatibility' groups)
-New procedures to prevent divergence and coordinate development across platforms (reserving flags, wider discussion before names for features/flags are decided)
-The monthly ZFS Leadership Call


Then switch gears and cover FreeBSD specific issues:

-The switch to ZoL as upstream
-Why we are making the change
-What we get out of it
-How it is better for all of OpenZFS
-Status report
-ZoL is OpenZFS, not Linux
-There is no LinuxKPI in ZFS (kill the FUD)
-What has OpenZFS done for me lately


Then an overview of upcoming changes and features in ZFS.
https://www.bsdcan.org/2019/schedule/ev ... 60.en.html

https://www.youtube.com/watch?v=E8kAVmvlBLQ

:'
Last edited by Moogle Stiltzkin on Sat Oct 19, 2019 12:44 pm, edited 1 time in total.
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

Post Reply

Return to “Presales”