crashing QNAP TS-h1277XU-RP-3700X-128G

Don't miss a thing. Post your questions and discussion about other uncategorized NAS features here.
Post Reply
anton.me
New here
Posts: 3
Joined: Thu Sep 24, 2020 6:09 pm

crashing QNAP TS-h1277XU-RP-3700X-128G

Post by anton.me » Thu Sep 24, 2020 6:39 pm

Hi,

We are running Samba, NFS, Virtualization Station and ContainerStation on a TS-h1277XU-RP-3700X-128G.

A couple of times the device has just stopped responding on NFS and all Virtual machines are unresponsive.

Seems zfs has crashed? Any one have any idea how to fix this or has similar problems?
Restart from web-admin does not work (just hanged in System Reboot, waiting to be stopped).

Here is the last lines of dmsg from ssh:

Code: Select all

[778810.176354] nfsd_buffered_filldir filtered @Recently-Snapshot
[810845.249031] flush_memory.sh (18699): drop_caches: 3
[818475.969910] Non ZBT_MICRO, ZBT_HEADER zap found!!!! db->db_data ffff88147106b600 value=3267436641439981603
[818475.979651] Non ZBT_MICRO, ZBT_HEADER zap found!!!! db->db_data ffff88147106b600 value=3267436641439981603
[818568.092324] Non ZBT_MICRO, ZBT_HEADER zap found!!!! db->db_data ffff88147106c000 value=0
[818568.100506] Non ZBT_MICRO, ZBT_HEADER zap found!!!! db->db_data ffff88147106c000 value=0
[818568.183348] Non ZBT_MICRO, ZBT_HEADER zap found!!!! db->db_data ffff88147106ba00 value=505814944114081882
[818568.193002] Non ZBT_MICRO, ZBT_HEADER zap found!!!! db->db_data ffff88147106ba00 value=505814944114081882
[818680.605872] cache_from_obj: Wrong slab cache. zio_buf_512 but object is from arc_buf_hdr_t
[818680.614219] ------------[ cut here ]------------
[818680.618933] WARNING: CPU: 7 PID: 4776 at mm/slab.h:377 kmem_cache_free+0xa5/0xc0
[818680.626406] Modules linked in: xt_conntrack xt_ipvs ip_vs_rr ip_vs_ftp ip_vs xt_nat xt_addrtype vfio_iommu_type1 vhost_scsi target_core_mod vhost_net vhost macvtap macvlan tap tun virtio_scsi virtio_pci virtio_net virtio_mmio virtio_console virtio_blk virtio_balloon virtio_rng virtio_ring virtio kvm_amd kvm fbdisk(O) rfcomm ib_iser(O) rdma_cm(O) ib_cm(O) iw_cm(O) bnxt_re(O) mlx5_ib(O) mlx4_ib(O) ib_core(O) iscsi_tcp(O) libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O) qla2xxx_qzsttgt(O) zscst_vdisk(O) iscsi_scst(O) scst(O) dummy br_netfilter bridge stp bonding xt_connmark xt_TCPMSS xt_LOG xt_set ip_set_hash_netiface ip_set_hash_net ip_set ipt_MASQUERADE xt_REDIRECT nf_nat_redirect iptable_nat nf_nat_masquerade_ipv4 nf_nat_ipv4 nf_nat xt_policy xt_mark 8021q ipv6 uvcvideo videobuf2_v4l2 videobuf2_vmalloc
[818680.697670]  videobuf2_memops videobuf2_core snd_usb_caiaq snd_usb_audio snd_usbmidi_lib snd_seq_midi snd_rawmidi fnotify(O) nfsd udf isofs sp5100_tco kcopy(PO) qtweak(PO) vfio_pci irqbypass vfio_virqfd vfio ufsd(PO) jnl(O) cdc_acm pl2303 usbserial qm2_i2c(O) zfs(O) icp(PO) lpl(O) drbd lru_cache flashcache(O) dm_tier_hro_algo dm_thin_pool dm_bio_prison dm_persistent_data hal_netlink(O) k10temp mlx5_core(O) mlx4_en(O) mlx4_core(O) mlx_compat(O) ixgbe mdio r8152 usbnet mii igb e1000e(O) mv14xx(PO) mpt3sas scsi_transport_sas raid_class qla2xxx_qzst(O) scsi_transport_fc uas usb_storage xhci_pci xhci_hcd usblp uhci_hcd ehci_pci ehci_hcd
[818680.753592] CPU: 7 PID: 4776 Comm: sa_db_rele_task Tainted: P           O    4.14.24-qnap #1
[818680.762102] Hardware name: Default string Default string/Default string, BIOS QZ54AR12 04/13/2020
[818680.771048] task: ffff881f6f2e22c0 task.stack: ffffc900009d8000
[818680.777053] RIP: 0010:kmem_cache_free+0xa5/0xc0
[818680.781670] RSP: 0018:ffffc900009dbc50 EFLAGS: 00010282
[818680.786974] RAX: 000000000000004e RBX: ffff881f6f15cd00 RCX: 0000000000000000
[818680.794187] RDX: ffff88200e9dc240 RSI: ffff88200e9d5558 RDI: ffff88200e9d5558
[818680.801397] RBP: ffff881471070000 R08: ffffc90000583000 R09: 0000000000000c41
[818680.808609] R10: ffff88120add4b98 R11: 0000000000000000 R12: ffff881be3482c10
[818680.815825] R13: ffff881846fb8380 R14: ffff881f87bac790 R15: ffff881f5389d000
[818680.823034] FS:  0000000000000000(0000) GS:ffff88200e9c0000(0000) knlGS:0000000000000000
[818680.831199] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[818680.837025] CR2: 00007f69d7208000 CR3: 0000001dbe63c000 CR4: 00000000003406e0
[818680.844412] Call Trace:
[818680.846966]  dbuf_clear+0x14e/0x160 [zfs]
[818680.851070]  dbuf_evict+0x9/0x20 [zfs]
[818680.854918]  dnode_destroy+0xf6/0x200 [zfs]
[818680.859208]  dnode_buf_pageout+0x35/0xb0 [zfs]
[818680.863742]  dbuf_rele_and_unlock+0x13d/0x520 [zfs]
[818680.868704]  ? update_curr+0x93/0xd0
[818680.872363]  ? dequeue_entity+0x562/0x810
[818680.876457]  ? mutex_lock+0x9/0x30
[818680.879956]  ? dnode_rele+0x61/0x90 [zfs]
[818680.884057]  ? mutex_lock+0x9/0x30
[818680.887552]  dbuf_rele_and_unlock+0x30e/0x520 [zfs]
[818680.892516]  ? __wake_up_common_lock+0x72/0x90
[818680.897047]  ? mutex_lock+0x9/0x30
[818680.900535]  taskq_thread+0x2ae/0x5a0 [lpl]
[818680.904801]  ? wake_up_q+0x60/0x60
[818680.908292]  ? taskq_thread_spawn+0x50/0x50 [lpl]
[818680.913075]  kthread+0x10a/0x140
[818680.916387]  ? __kthread_parkme+0x70/0x70
[818680.920484]  ? call_usermodehelper_exec_async+0x12d/0x140
[818680.925964]  ret_from_fork+0x22/0x40
[818680.929632] Code: 74 33 48 3b 98 80 00 00 00 48 89 c7 74 90 48 8b 48 58 48 c7 c6 e8 46 e1 81 48 c7 c7 60 55 fd 81 31 c0 48 8b 53 58 e8 3c f7 f4 ff <0f> 0b 48 89 df e9 69 ff ff ff 48 89 df e9 61 ff ff ff 66 0f 1f 
[818680.948549] ---[ end trace 3c55df0a6e03343d ]---
[818695.067585] cache_from_obj: Wrong slab cache. zio_buf_512 but object is from arc_buf_hdr_t
[818719.189403] cache_from_obj: Wrong slab cache. zio_buf_512 but object is from arc_buf_hdr_t
[839412.130446] nfsd_buffered_filldir filtered @Recently-Snapshot
[846763.414487] nfsd_buffered_filldir filtered @Recently-Snapshot
[848867.991935] BUG: unable to handle kernel paging request at 000000005f632e54
[848867.998994] IP: buf_hash_find+0xba/0x130 [zfs]
[848868.003523] PGD 0 P4D 0 
[848868.006152] Oops: 0000 [#1] SMP NOPTI
[848868.009896] Modules linked in: xt_conntrack xt_ipvs ip_vs_rr ip_vs_ftp ip_vs xt_nat xt_addrtype vfio_iommu_type1 vhost_scsi target_core_mod vhost_net vhost macvtap macvlan tap tun virtio_scsi virtio_pci virtio_net virtio_mmio virtio_console virtio_blk virtio_balloon virtio_rng virtio_ring virtio kvm_amd kvm fbdisk(O) rfcomm ib_iser(O) rdma_cm(O) ib_cm(O) iw_cm(O) bnxt_re(O) mlx5_ib(O) mlx4_ib(O) ib_core(O) iscsi_tcp(O) libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O) qla2xxx_qzsttgt(O) zscst_vdisk(O) iscsi_scst(O) scst(O) dummy br_netfilter bridge stp bonding xt_connmark xt_TCPMSS xt_LOG xt_set ip_set_hash_netiface ip_set_hash_net ip_set ipt_MASQUERADE xt_REDIRECT nf_nat_redirect iptable_nat nf_nat_masquerade_ipv4 nf_nat_ipv4 nf_nat xt_policy xt_mark 8021q ipv6 uvcvideo videobuf2_v4l2 videobuf2_vmalloc
[848868.081169]  videobuf2_memops videobuf2_core snd_usb_caiaq snd_usb_audio snd_usbmidi_lib snd_seq_midi snd_rawmidi fnotify(O) nfsd udf isofs sp5100_tco kcopy(PO) qtweak(PO) vfio_pci irqbypass vfio_virqfd vfio ufsd(PO) jnl(O) cdc_acm pl2303 usbserial qm2_i2c(O) zfs(O) icp(PO) lpl(O) drbd lru_cache flashcache(O) dm_tier_hro_algo dm_thin_pool dm_bio_prison dm_persistent_data hal_netlink(O) k10temp mlx5_core(O) mlx4_en(O) mlx4_core(O) mlx_compat(O) ixgbe mdio r8152 usbnet mii igb e1000e(O) mv14xx(PO) mpt3sas scsi_transport_sas raid_class qla2xxx_qzst(O) scsi_transport_fc uas usb_storage xhci_pci xhci_hcd usblp uhci_hcd ehci_pci ehci_hcd
[848868.137105] CPU: 6 PID: 9013 Comm: sync zpool1 Tainted: P        W  O    4.14.24-qnap #1
[848868.145272] Hardware name: Default string Default string/Default string, BIOS QZ54AR12 04/13/2020
[848868.154215] task: ffff881f8544e500 task.stack: ffffc9002e37c000
[848868.160236] RIP: 0010:buf_hash_find+0xba/0x130 [zfs]
[848868.165282] RSP: 0018:ffffc9002e37f8b0 EFLAGS: 00010202
[848868.170590] RAX: 000000005f632e54 RBX: 0000000000450324 RCX: 0000000000000024
[848868.177802] RDX: 0000000000000160 RSI: 0000000000001280 RDI: ffffffffa092c680
[848868.185015] RBP: ffffc9002e37f8f8 R08: 12657fa4c3f06507 R09: 0000000000000000
[848868.192229] R10: 0000000000000001 R11: 0000000000214004 R12: ffff881442bfa608
[848868.199443] R13: 70200390a50d2262 R14: ffffffffa092c680 R15: 0000000000106b01
[848868.206658] FS:  0000000000000000(0000) GS:ffff88200e980000(0000) knlGS:0000000000000000
[848868.214829] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[848868.220658] CR2: 000000005f632e54 CR3: 0000001c72c5c000 CR4: 00000000003406e0
[848868.227867] Call Trace:
[848868.230422]  arc_freed+0x29/0xd0 [zfs]
[848868.234273]  zio_free_sync+0x50/0x100 [zfs]
[848868.238552]  zio_free+0x9a/0x100 [zfs]
[848868.242396]  zil_sync+0x1c1/0x4a0 [zfs]
[848868.246332]  ? dnode_sync+0x59a/0xb00 [zfs]
[848868.250609]  ? avl_find+0x4c/0x80 [zfs]
[848868.254551]  dmu_objset_sync+0x1fb/0x400 [zfs]
[848868.259095]  dsl_dataset_sync+0x3b/0xe0 [zfs]
[848868.263545]  dsl_pool_sync+0x135/0xe40 [zfs]
[848868.267918]  ? zap_lookup+0xd/0x20 [zfs]
[848868.271940]  spa_sync+0x8ed/0x4810 [zfs]
[848868.275950]  ? try_to_wake_up+0x1cc/0x350
[848868.280047]  ? autoremove_wake_function+0x9/0x30
[848868.284753]  ? __wake_up_common+0x82/0x120
[848868.288954]  txg_sync_thread+0x473/0x620 [zfs]
[848868.293503]  ? txg_quiesce_thread+0xc30/0xc30 [zfs]
[848868.298469]  ? __thread_exit+0x10/0x10 [lpl]
[848868.302824]  thread_generic_wrapper+0x7a/0xc0 [lpl]
[848868.307786]  kthread+0x10a/0x140
[848868.311102]  ? __kthread_parkme+0x70/0x70
[848868.315201]  ret_from_fork+0x22/0x40
[848868.318863] Code: 48 8b 4c 24 08 48 89 86 28 b4 92 a0 48 8b 05 b6 f9 2f 00 48 8b 04 d8 48 85 c0 74 34 49 8b 14 24 eb 09 48 8b 40 60 48 85 c0 74 25 <48> 39 10 75 f2 49 8b 7c 24 08 48 39 78 08 75 e7 4c 39 78 10 75 
[848868.337799] RIP: buf_hash_find+0xba/0x130 [zfs] RSP: ffffc9002e37f8b0
[848868.344320] CR2: 000000005f632e54
[848868.347719] ---[ end trace 3c55df0a6e03343e ]---
[848898.245952] slow spa_sync: pool zpool1 txg 1075969 pass 1 started 30 seconds ago, calls 1
[848900.805955] slow spa_sync: pool zpool2 txg 539158 pass 1 started 30 seconds ago, calls 3


User avatar
Trexx
Ask me anything
Posts: 5291
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: crashing QNAP TS-h1277XU-RP-3700X-128G

Post by Trexx » Thu Sep 24, 2020 7:49 pm

I would suggest opening a Helpdesk ticket with QNAP support as there is very limited experience with QuTS Hero here at this time.


Sent from my iPad using Tapatalk
Paul

Model: TS-877-1600 FW: 4.4.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x 500GB Evo 860
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
GPU: EVGA GTX 1060 6GB
UPS: CP AVR1350

Model:TVS-673 32GB FW: 4.4.3.x Test/Backup Box
Model:TS-228a FW: 4.4.3.x Test/Backup Box
-----------------------------------------------------------------------------------------------------------------------------------------
NAS RAID Rebuild Times | Live QTS Videos | | QNAP NAS Guide | Information needed when you ask for HELP | QNAP Links, Tutorials, etc.
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq

anton.me
New here
Posts: 3
Joined: Thu Sep 24, 2020 6:09 pm

Re: crashing QNAP TS-h1277XU-RP-3700X-128G

Post by anton.me » Thu Sep 24, 2020 7:55 pm

Thank you Paul,

Thats the downside of buying brand new models i guess :)
Great idea. If anything comes from that i'll post it here for reference.

Best,
Anton

anton.me
New here
Posts: 3
Joined: Thu Sep 24, 2020 6:09 pm

Re: crashing QNAP TS-h1277XU-RP-3700X-128G

Post by anton.me » Mon Oct 05, 2020 2:59 pm

Short update: we have disabled shared-memory for the qemu virtual-station. System seems more stable now.

Before we had “blocked for more than 120 seconds” from our Linux kernels on the VMs. Have not seen it lately.

Dont know if this was the solution.

Post Reply

Return to “Miscellaneous”