<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Subject: xen-hypervisor-4.11-amd64: DomU freeze with
linux-image-4.9.0-8-amd64 and gnttab_max_frames=256
<br>
Package: xen-hypervisor-4.11-amd64
<br>
Version: 4.11.1-1
<br>
Justification: breaks the whole system
<br>
Severity: critical
<br>
Tags: upstream
<br>
<br>
Dear Maintainers,
<br>
<br>
The issue I have looks a lot to the bug reported here :
<a class="moz-txt-link-freetext" href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880554">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880554</a>
<br>
except that putting gnttab_max_frames=256 didn't solved my issue
and the environement may be a bit different.
<br>
<br>
So here is the situation,
<br>
<br>
I've downloaded Xen sources from the repo, compiled them with
XSM/Flask support, made deb packages and installed them on a fresh
install of Debian 9.
<br>
<br>
Dom0 4-6 vcpu with 4G ram
<br>
<br>
Dom0 and DomU uname -a :
<br>
Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64
GNU/Linux
<br>
<br>
DomUs are launched with PvGrub and
seclabel='system_u:system_r:isolated_domU_t'
<br>
<br>
The problematic DomU is connected to a LUN target (iscsi) for some
storage and run some docker containers. (just in case ...)
<br>
<br>
I thought it might be related to the iscsi drive so I completely
uninstalled it but the problem is still occuring.
<br>
<br>
DomU.conf :
<br>
<br>
kernel = '/usr/lib/grub-xen/grub-x86_64-xen.bin'
<br>
extra = '(hd0)/boot/grub/grub.cfg'
<br>
<br>
seclabel='system_u:system_r:isolated_domU_t'
<br>
<br>
vcpus = '4'
<br>
memory = '4048'
<br>
<br>
root = '/dev/xvda2 ro'
<br>
disk = [
<br>
'phy:/dev/vg/domu-root,xvda2,w',
<br>
'phy:/dev/vg/domu-var-lib-docker,xvda10,w',
<br>
'phy:/dev/vg/domu-var-log-audit,xvda9,w',
<br>
'phy:/dev/vg/domu-var-log,xvda8,w',
<br>
'phy:/dev/vg/domu-home,xvda7,w',
<br>
'phy:/dev/vg/domu-var,xvda6,w',
<br>
'phy:/dev/vg/domu-usr,xvda5,w',
<br>
'phy:/dev/vg/domu-tmp,xvda4,w',
<br>
'phy:/dev/vg/domu-srv,xvda3,w',
<br>
'phy:/dev/vg/domu-swap,xvda1,w',
<br>
'phy:/dev/vg/domu-storage,xvda11,w',
<br>
]
<br>
<br>
name = 'domu'
<br>
<br>
vif = [ 'mac=xx:xx:xx:xx:xx:xx, bridge=xenbr0' ]
<br>
<br>
on_poweroff = 'destroy'
<br>
on_reboot = 'restart'
<br>
on_crash = 'restart'
<br>
<br>
<br>
<br>
<br>
After some "heavy" IO loads, the DomU freeze and the console show
messages like following :
<br>
<br>
<br>
<font size="-1">[ 484.316142] INFO: task kworker/u8:1:48 blocked
for more than 120 seconds.
<br>
[ 484.316183] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 484.316208] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 484.316240] kworker/u8:1 D 0 48 2 0x00000080
<br>
[ 484.316276] Workqueue: writeback wb_workfn (flush-202:4)
<br>
[ 484.316313] ffff8800e8252540 0000000000000000
ffff8800f2c822c0 ffff8800f7f98980
<br>
[ 484.316330] ffff8800f6c06380 ffffc900408d35d0
ffffffff816144b9 0000000000000001
<br>
[ 484.316347] 00ff8800ebbf60c0 ffff8800f7f98980
ffffffff813135d9 ffff8800f2c822c0
<br>
[ 484.316367] Call Trace:
<br>
[ 484.316378] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 484.316389] [<ffffffff813135d9>] ?
blk_mq_flush_plug_list+0x139/0x160
<br>
[ 484.316401] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 484.316410] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 484.316421] [<ffffffff813086b3>] ?
blk_flush_plug_list+0xc3/0x230
<br>
[ 484.316433] [<ffffffff811861b9>] ?
mempool_alloc+0x69/0x190
<br>
[ 484.316444] [<ffffffff8101bf91>] ?
xen_clocksource_get_cycles+0x11/0x20
<br>
[ 484.316455] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 484.316467] [<ffffffff81363964>] ?
__sbitmap_queue_get+0x24/0x90
<br>
[ 484.316477] [<ffffffff81313b80>] ?
bt_get.isra.6+0x160/0x220
<br>
[ 484.316489] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 484.316500] [<ffffffff81313ef3>] ?
blk_mq_get_tag+0x23/0x90
<br>
[ 484.316510] [<ffffffff8130f77a>] ?
__blk_mq_alloc_request+0x1a/0x220
<br>
[ 484.316520] [<ffffffff81310649>] ?
blk_mq_map_request+0xd9/0x170
<br>
[ 484.316531] [<ffffffff81312e8b>] ?
blk_mq_make_request+0xbb/0x580
<br>
[ 484.316541] [<ffffffff81306d35>] ?
generic_make_request+0x115/0x2d0
<br>
[ 484.316551] [<ffffffff81306f66>] ?
submit_bio+0x76/0x140
<br>
[ 484.316573] [<ffffffffc01be358>] ?
ext4_io_submit+0x48/0x60 [ext4]
<br>
[ 484.316593] [<ffffffffc01be5c6>] ?
ext4_bio_write_page+0x236/0x4d0 [ext4]
<br>
[ 484.316605] [<ffffffff81190f98>] ?
clear_page_dirty_for_io+0x148/0x1b0
<br>
[ 484.316621] [<ffffffffc01b3a32>] ?
mpage_submit_page+0x52/0x70 [ext4]
<br>
[ 484.316636] [<ffffffffc01b3c83>] ?
mpage_map_and_submit_buffers+0x133/0x230 [ext4]
<br>
[ 484.316658] [<ffffffffc01b8fb0>] ?
ext4_writepages+0x7c0/0xd60 [ext4]
<br>
[ 484.316669] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 484.316680] [<ffffffff810e9c57>] ?
mod_timer+0x177/0x3b0
<br>
[ 484.316690] [<ffffffff8123a7bd>] ?
__writeback_single_inode+0x3d/0x320
<br>
[ 484.316700] [<ffffffff8123af61>] ?
writeback_sb_inodes+0x221/0x4f0
<br>
[ 484.316714] [<ffffffff8123b2b7>] ?
__writeback_inodes_wb+0x87/0xb0
<br>
[ 484.316724] [<ffffffff8123b62e>] ?
wb_writeback+0x27e/0x310
<br>
[ 484.316734] [<ffffffff8123be5e>] ?
wb_workfn+0x17e/0x380
<br>
[ 484.316745] [<ffffffff8109427a>] ?
process_one_work+0x18a/0x430
<br>
[ 484.316755] [<ffffffff8109456d>] ?
worker_thread+0x4d/0x490
<br>
[ 484.316765] [<ffffffff81094520>] ?
process_one_work+0x430/0x430
<br>
[ 484.316775] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 484.316786] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 484.316796] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 484.316808] INFO: task jbd2/xvda5-8:194 blocked for more than
120 seconds.
<br>
[ 484.316819] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 484.316828] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 484.316841] jbd2/xvda5-8 D 0 194 2 0x00000080
<br>
[ 484.316851] ffff8800f2dac500 0000000000000000
ffff88000298c380 ffff8800f7e18980
<br>
[ 484.316867] ffff8800f2dfa7c0 ffffc9004093bb30
ffffffff816144b9 ffffc9004093bbe8
<br>
[ 484.316885] 00ffffff81307eaf ffff8800f7e18980
0000000000000000 ffff88000298c380
<br>
[ 484.316903] Call Trace:
<br>
[ 484.316910] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 484.316920] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.316930] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 484.316938] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 484.316952] [<ffffffff810f107e>] ?
ktime_get+0x3e/0xb0
<br>
[ 484.316961] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.316971] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 484.316980] [<ffffffff810bcf87>] ?
prepare_to_wait+0x57/0x80
<br>
[ 484.316991] [<ffffffff81615187>] ?
bit_wait_io+0x17/0x60
<br>
[ 484.317000] [<ffffffff81614d45>] ?
__wait_on_bit+0x55/0x80
<br>
[ 484.317009] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.317018] [<ffffffff81614eae>] ?
out_of_line_wait_on_bit+0x7e/0xa0
<br>
[ 484.317029] [<ffffffff810bd3f0>] ?
wake_atomic_t_function+0x60/0x60
<br>
[ 484.317043] [<ffffffffc0076e6e>] ?
jbd2_journal_commit_transaction+0xdde/0x17b0 [jbd2]
<br>
[ 484.317059] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 484.317069] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 484.317080] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 484.317090] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 484.317102] [<ffffffffc007bc62>] ?
kjournald2+0xc2/0x260 [jbd2]
<br>
[ 484.317114] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 484.317125] [<ffffffffc007bba0>] ?
commit_timeout+0x10/0x10 [jbd2]
<br>
[ 484.317135] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 484.317146] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 484.317154] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 484.317164] INFO: task jbd2/xvda4-8:328 blocked for more than
120 seconds.
<br>
[ 484.317176] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 484.317185] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 484.317196] jbd2/xvda4-8 D 0 328 2 0x00000080
<br>
[ 484.317205] ffff8800f5c3b200 0000000000000000
ffff8800f2d56800 ffff8800f7f18980
<br>
[ 484.317221] ffff880002cb0480 ffffc900409c7a70
ffffffff816144b9 ffffffff81092a28
<br>
[ 484.317237] 00ffe8ffffd8b540 ffff8800f7f18980
ffffffff813135d9 ffff8800f2d56800
<br>
[ 484.317253] Call Trace:
<br>
[ 484.317260] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 484.317268] [<ffffffff81092a28>] ?
queue_work_on+0x28/0x40
<br>
[ 484.317278] [<ffffffff813135d9>] ?
blk_mq_flush_plug_list+0x139/0x160
<br>
[ 484.317287] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.317296] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 484.317304] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 484.317315] [<ffffffff8101bf91>] ?
xen_clocksource_get_cycles+0x11/0x20
<br>
[ 484.317325] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.317334] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 484.317343] [<ffffffff810bcf87>] ?
prepare_to_wait+0x57/0x80
<br>
[ 484.317353] [<ffffffff81615187>] ?
bit_wait_io+0x17/0x60
<br>
[ 484.317362] [<ffffffff81614d45>] ?
__wait_on_bit+0x55/0x80
<br>
[ 484.317373] [<ffffffff811828c8>] ?
find_get_pages_tag+0x158/0x2e0
<br>
[ 484.317383] [<ffffffff81181b2f>] ?
wait_on_page_bit+0x7f/0xa0
<br>
[ 484.317393] [<ffffffff810bd3f0>] ?
wake_atomic_t_function+0x60/0x60
<br>
[ 484.317403] [<ffffffff81181c30>] ?
__filemap_fdatawait_range+0xe0/0x140
<br>
[ 484.317413] [<ffffffff81181c9f>] ?
filemap_fdatawait_range+0xf/0x30
<br>
[ 484.317425] [<ffffffffc00767cd>] ?
jbd2_journal_commit_transaction+0x73d/0x17b0 [jbd2]
<br>
[ 484.317437] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 484.317447] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 484.317456] [<ffffffff810158e2>] ?
xen_load_sp0+0x82/0x170
<br>
[ 484.317465] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 484.317475] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 484.317486] [<ffffffffc007bc62>] ?
kjournald2+0xc2/0x260 [jbd2]
<br>
[ 484.317496] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 484.516588] [<ffffffffc007bba0>] ?
commit_timeout+0x10/0x10 [jbd2]
<br>
[ 484.516603] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 484.516612] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 484.516624] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 484.516640] INFO: task jbd2/xvda8-8:367 blocked for more than
120 seconds.
<br>
[ 484.516656] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 484.516667] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 484.516684] jbd2/xvda8-8 D 0 367 2 0x00000080
<br>
[ 484.516704] ffff8800f5c3b200 0000000000000000
ffff8800f5cac740 ffff8800f7f18980
<br>
[ 484.516741] ffff880002cb0480 ffffc90040ae7b30
ffffffff816144b9 ffff8800ee926000
<br>
[ 484.516776] 00ff8800ee943200 ffff8800f7f18980
ffff88008e403f00 ffff8800f5cac740
<br>
[ 484.516794] Call Trace:
<br>
[ 484.516802] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 484.516811] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.516823] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 484.516831] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 484.516841] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 484.516850] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 484.516861] [<ffffffff8101bf91>] ?
xen_clocksource_get_cycles+0x11/0x20
<br>
[ 484.516871] [<ffffffff810f107e>] ?
ktime_get+0x3e/0xb0
<br>
[ 484.516880] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.516889] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 484.516899] [<ffffffff810bcf87>] ?
prepare_to_wait+0x57/0x80
<br>
[ 484.516910] [<ffffffff81615187>] ?
bit_wait_io+0x17/0x60
<br>
[ 484.516921] [<ffffffff81614d45>] ?
__wait_on_bit+0x55/0x80
<br>
[ 484.516930] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 484.516940] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 484.516950] [<ffffffff81614eae>] ?
out_of_line_wait_on_bit+0x7e/0xa0
<br>
[ 484.516961] [<ffffffff810bd3f0>] ?
wake_atomic_t_function+0x60/0x60
<br>
[ 484.516972] [<ffffffffc0076fe5>] ?
jbd2_journal_commit_transaction+0xf55/0x17b0 [jbd2]
<br>
[ 484.516985] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 484.516994] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 484.517005] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 484.517016] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 484.517028] [<ffffffffc007bc62>] ?
kjournald2+0xc2/0x260 [jbd2]
<br>
[ 484.517038] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 484.517051] [<ffffffffc007bba0>] ?
commit_timeout+0x10/0x10 [jbd2]
<br>
[ 484.517062] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 484.517071] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 484.517079] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 605.148081] INFO: task kworker/u8:1:48 blocked for more than
120 seconds.
<br>
[ 605.148121] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 605.148146] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 605.148178] kworker/u8:1 D 0 48 2 0x00000080
<br>
[ 605.148214] Workqueue: writeback wb_workfn (flush-202:4)
<br>
[ 605.148242] ffff8800e8252540 0000000000000000
ffff8800f2c822c0 ffff8800f7f98980
<br>
[ 605.148287] ffff8800f6c06380 ffffc900408d35d0
ffffffff816144b9 0000000000000001
<br>
[ 605.148336] 00ff8800ebbf60c0 ffff8800f7f98980
ffffffff813135d9 ffff8800f2c822c0
<br>
[ 605.148381] Call Trace:
<br>
[ 605.148410] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 605.148438] [<ffffffff813135d9>] ?
blk_mq_flush_plug_list+0x139/0x160
<br>
[ 605.148476] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 605.148493] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 605.148510] [<ffffffff813086b3>] ?
blk_flush_plug_list+0xc3/0x230
<br>
[ 605.148527] [<ffffffff811861b9>] ?
mempool_alloc+0x69/0x190
<br>
[ 605.148541] [<ffffffff8101bf91>] ?
xen_clocksource_get_cycles+0x11/0x20
<br>
[ 605.148555] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 605.148571] [<ffffffff81363964>] ?
__sbitmap_queue_get+0x24/0x90
<br>
[ 605.148586] [<ffffffff81313b80>] ?
bt_get.isra.6+0x160/0x220
<br>
[ 605.148602] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 605.148618] [<ffffffff81313ef3>] ?
blk_mq_get_tag+0x23/0x90
<br>
[ 605.148629] [<ffffffff8130f77a>] ?
__blk_mq_alloc_request+0x1a/0x220
<br>
[ 605.148639] [<ffffffff81310649>] ?
blk_mq_map_request+0xd9/0x170
<br>
[ 605.148649] [<ffffffff81312e8b>] ?
blk_mq_make_request+0xbb/0x580
<br>
[ 605.148660] [<ffffffff81306d35>] ?
generic_make_request+0x115/0x2d0
<br>
[ 605.148670] [<ffffffff81306f66>] ?
submit_bio+0x76/0x140
<br>
[ 605.148694] [<ffffffffc01be358>] ?
ext4_io_submit+0x48/0x60 [ext4]
<br>
[ 605.148714] [<ffffffffc01be5c6>] ?
ext4_bio_write_page+0x236/0x4d0 [ext4]
<br>
[ 605.148725] [<ffffffff81190f98>] ?
clear_page_dirty_for_io+0x148/0x1b0
<br>
[ 605.148742] [<ffffffffc01b3a32>] ?
mpage_submit_page+0x52/0x70 [ext4]
<br>
[ 605.148757] [<ffffffffc01b3c83>] ?
mpage_map_and_submit_buffers+0x133/0x230 [ext4]
<br>
[ 605.148777] [<ffffffffc01b8fb0>] ?
ext4_writepages+0x7c0/0xd60 [ext4]
<br>
[ 605.148788] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 605.148799] [<ffffffff810e9c57>] ?
mod_timer+0x177/0x3b0
<br>
[ 605.148809] [<ffffffff8123a7bd>] ?
__writeback_single_inode+0x3d/0x320
<br>
[ 605.148819] [<ffffffff8123af61>] ?
writeback_sb_inodes+0x221/0x4f0
<br>
[ 605.148829] [<ffffffff8123b2b7>] ?
__writeback_inodes_wb+0x87/0xb0
<br>
[ 605.148840] [<ffffffff8123b62e>] ?
wb_writeback+0x27e/0x310
<br>
[ 605.148850] [<ffffffff8123be5e>] ?
wb_workfn+0x17e/0x380
<br>
[ 605.148859] [<ffffffff8109427a>] ?
process_one_work+0x18a/0x430
<br>
[ 605.148871] [<ffffffff8109456d>] ?
worker_thread+0x4d/0x490
<br>
[ 605.148881] [<ffffffff81094520>] ?
process_one_work+0x430/0x430
<br>
[ 605.148891] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 605.148900] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 605.148909] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 605.148920] INFO: task jbd2/xvda5-8:194 blocked for more than
120 seconds.
<br>
[ 605.148929] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 605.148941] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 605.148952] jbd2/xvda5-8 D 0 194 2 0x00000080
<br>
[ 605.148961] ffff8800f2dac500 0000000000000000
ffff88000298c380 ffff8800f7e18980
<br>
[ 605.148979] ffff8800f2dfa7c0 ffffc9004093bb30
ffffffff816144b9 ffffc9004093bbe8
<br>
[ 605.148995] 00ffffff81307eaf ffff8800f7e18980
0000000000000000 ffff88000298c380
<br>
[ 605.149011] Call Trace:
<br>
[ 605.149019] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 605.149028] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.149038] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 605.149046] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 605.149057] [<ffffffff810f107e>] ?
ktime_get+0x3e/0xb0
<br>
[ 605.149066] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.149075] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 605.149085] [<ffffffff810bcf87>] ?
prepare_to_wait+0x57/0x80
<br>
[ 605.149096] [<ffffffff81615187>] ?
bit_wait_io+0x17/0x60
<br>
[ 605.149106] [<ffffffff81614d45>] ?
__wait_on_bit+0x55/0x80
<br>
[ 605.149115] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.149124] [<ffffffff81614eae>] ?
out_of_line_wait_on_bit+0x7e/0xa0
<br>
[ 605.149134] [<ffffffff810bd3f0>] ?
wake_atomic_t_function+0x60/0x60
<br>
[ 605.149150] [<ffffffffc0076e6e>] ?
jbd2_journal_commit_transaction+0xdde/0x17b0 [jbd2]
<br>
[ 605.149162] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 605.149172] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 605.149184] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 605.149194] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 605.149206] [<ffffffffc007bc62>] ?
kjournald2+0xc2/0x260 [jbd2]
<br>
[ 605.149216] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 605.149227] [<ffffffffc007bba0>] ?
commit_timeout+0x10/0x10 [jbd2]
<br>
[ 605.149237] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 605.149246] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 605.149254] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 605.149264] INFO: task jbd2/xvda4-8:328 blocked for more than
120 seconds.
<br>
[ 605.149273] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 605.149283] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 605.149294] jbd2/xvda4-8 D 0 328 2 0x00000080
<br>
[ 605.149303] ffff8800f5c3b200 0000000000000000
ffff8800f2d56800 ffff8800f7f18980
<br>
[ 605.149320] ffff880002cb0480 ffffc900409c7a70
ffffffff816144b9 ffffffff81092a28
<br>
[ 605.149337] 00ffe8ffffd8b540 ffff8800f7f18980
ffffffff813135d9 ffff8800f2d56800
<br>
[ 605.149354] Call Trace:
<br>
[ 605.149361] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 605.149370] [<ffffffff81092a28>] ?
queue_work_on+0x28/0x40
<br>
[ 605.149379] [<ffffffff813135d9>] ?
blk_mq_flush_plug_list+0x139/0x160
<br>
[ 605.149389] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.149397] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 605.149406] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 605.149416] [<ffffffff8101bf91>] ?
xen_clocksource_get_cycles+0x11/0x20
<br>
[ 605.149426] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.149434] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 605.149444] [<ffffffff810bcf87>] ?
prepare_to_wait+0x57/0x80
<br>
[ 605.149454] [<ffffffff81615187>] ?
bit_wait_io+0x17/0x60
<br>
[ 605.149463] [<ffffffff81614d45>] ?
__wait_on_bit+0x55/0x80
<br>
[ 605.149473] [<ffffffff811828c8>] ?
find_get_pages_tag+0x158/0x2e0
<br>
[ 605.149483] [<ffffffff81181b2f>] ?
wait_on_page_bit+0x7f/0xa0
<br>
[ 605.149493] [<ffffffff810bd3f0>] ?
wake_atomic_t_function+0x60/0x60
<br>
[ 605.149503] [<ffffffff81181c30>] ?
__filemap_fdatawait_range+0xe0/0x140
<br>
[ 605.149513] [<ffffffff81181c9f>] ?
filemap_fdatawait_range+0xf/0x30
<br>
[ 605.149525] [<ffffffffc00767cd>] ?
jbd2_journal_commit_transaction+0x73d/0x17b0 [jbd2]
<br>
[ 605.149537] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 605.149546] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 605.149555] [<ffffffff810158e2>] ?
xen_load_sp0+0x82/0x170
<br>
[ 605.149565] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 605.149574] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 605.149585] [<ffffffffc007bc62>] ?
kjournald2+0xc2/0x260 [jbd2]
<br>
[ 605.149595] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 605.149606] [<ffffffffc007bba0>] ?
commit_timeout+0x10/0x10 [jbd2]
<br>
[ 605.149616] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 605.149624] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 605.149633] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 605.149642] INFO: task jbd2/xvda8-8:367 blocked for more than
120 seconds.
<br>
[ 605.348704] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 605.348715] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 605.348727] jbd2/xvda8-8 D 0 367 2 0x00000080
<br>
[ 605.348740] ffff8800f5c3b200 0000000000000000
ffff8800f5cac740 ffff8800f7f18980
<br>
[ 605.348779] ffff880002cb0480 ffffc90040ae7b30
ffffffff816144b9 ffff8800ee926000
<br>
[ 605.348820] 00ff8800ee943200 ffff8800f7f18980
ffff88008e403f00 ffff8800f5cac740
<br>
[ 605.348854] Call Trace:
<br>
[ 605.348866] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 605.348879] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.348900] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 605.348915] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 605.348929] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 605.348939] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 605.348950] [<ffffffff8101bf91>] ?
xen_clocksource_get_cycles+0x11/0x20
<br>
[ 605.348961] [<ffffffff810f107e>] ?
ktime_get+0x3e/0xb0
<br>
[ 605.348971] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.348980] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 605.348991] [<ffffffff810bcf87>] ?
prepare_to_wait+0x57/0x80
<br>
[ 605.349002] [<ffffffff81615187>] ?
bit_wait_io+0x17/0x60
<br>
[ 605.349013] [<ffffffff81614d45>] ?
__wait_on_bit+0x55/0x80
<br>
[ 605.349023] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 605.349037] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 605.349046] [<ffffffff81614eae>] ?
out_of_line_wait_on_bit+0x7e/0xa0
<br>
[ 605.349058] [<ffffffff810bd3f0>] ?
wake_atomic_t_function+0x60/0x60
<br>
[ 605.349070] [<ffffffffc0076fe5>] ?
jbd2_journal_commit_transaction+0xf55/0x17b0 [jbd2]
<br>
[ 605.349083] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 605.349093] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 605.349104] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 605.349115] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 605.349127] [<ffffffffc007bc62>] ?
kjournald2+0xc2/0x260 [jbd2]
<br>
[ 605.349139] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 605.349151] [<ffffffffc007bba0>] ?
commit_timeout+0x10/0x10 [jbd2]
<br>
[ 605.349162] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 605.349172] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 605.349181] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 725.980078] INFO: task kworker/u8:1:48 blocked for more than
120 seconds.
<br>
[ 725.980098] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 725.980107] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 725.980119] kworker/u8:1 D 0 48 2 0x00000080
<br>
[ 725.980135] Workqueue: writeback wb_workfn (flush-202:4)
<br>
[ 725.980146] ffff8800e8252540 0000000000000000
ffff8800f2c822c0 ffff8800f7f98980
<br>
[ 725.980164] ffff8800f6c06380 ffffc900408d35d0
ffffffff816144b9 0000000000000001
<br>
[ 725.980181] 00ff8800ebbf60c0 ffff8800f7f98980
ffffffff813135d9 ffff8800f2c822c0
<br>
[ 725.980199] Call Trace:
<br>
[ 725.980210] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 725.980221] [<ffffffff813135d9>] ?
blk_mq_flush_plug_list+0x139/0x160
<br>
[ 725.980232] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 725.980243] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 725.980254] [<ffffffff813086b3>] ?
blk_flush_plug_list+0xc3/0x230
<br>
[ 725.980266] [<ffffffff811861b9>] ?
mempool_alloc+0x69/0x190
<br>
[ 725.980277] [<ffffffff8101bf91>] ?
xen_clocksource_get_cycles+0x11/0x20
<br>
[ 725.980290] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 725.980300] [<ffffffff81363964>] ?
__sbitmap_queue_get+0x24/0x90
<br>
[ 725.980312] [<ffffffff81313b80>] ?
bt_get.isra.6+0x160/0x220
<br>
[ 725.980323] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 725.980335] [<ffffffff81313ef3>] ?
blk_mq_get_tag+0x23/0x90
<br>
[ 725.980346] [<ffffffff8130f77a>] ?
__blk_mq_alloc_request+0x1a/0x220
<br>
[ 725.980359] [<ffffffff81310649>] ?
blk_mq_map_request+0xd9/0x170
<br>
[ 725.980369] [<ffffffff81312e8b>] ?
blk_mq_make_request+0xbb/0x580
<br>
[ 725.980382] [<ffffffff81306d35>] ?
generic_make_request+0x115/0x2d0
<br>
[ 725.980392] [<ffffffff81306f66>] ?
submit_bio+0x76/0x140
<br>
[ 725.980415] [<ffffffffc01be358>] ?
ext4_io_submit+0x48/0x60 [ext4]
<br>
[ 725.980435] [<ffffffffc01be5c6>] ?
ext4_bio_write_page+0x236/0x4d0 [ext4]
<br>
[ 725.980448] [<ffffffff81190f98>] ?
clear_page_dirty_for_io+0x148/0x1b0
<br>
[ 725.980465] [<ffffffffc01b3a32>] ?
mpage_submit_page+0x52/0x70 [ext4]
<br>
[ 725.980480] [<ffffffffc01b3c83>] ?
mpage_map_and_submit_buffers+0x133/0x230 [ext4]
<br>
[ 725.980502] [<ffffffffc01b8fb0>] ?
ext4_writepages+0x7c0/0xd60 [ext4]
<br>
[ 725.980513] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 725.980523] [<ffffffff810e9c57>] ?
mod_timer+0x177/0x3b0
<br>
[ 725.980534] [<ffffffff8123a7bd>] ?
__writeback_single_inode+0x3d/0x320
<br>
[ 725.980545] [<ffffffff8123af61>] ?
writeback_sb_inodes+0x221/0x4f0
<br>
[ 725.980556] [<ffffffff8123b2b7>] ?
__writeback_inodes_wb+0x87/0xb0
<br>
[ 725.980567] [<ffffffff8123b62e>] ?
wb_writeback+0x27e/0x310
<br>
[ 725.980579] [<ffffffff8123be5e>] ?
wb_workfn+0x17e/0x380
<br>
[ 725.980588] [<ffffffff8109427a>] ?
process_one_work+0x18a/0x430
<br>
[ 725.980599] [<ffffffff8109456d>] ?
worker_thread+0x4d/0x490
<br>
[ 725.980609] [<ffffffff81094520>] ?
process_one_work+0x430/0x430
<br>
[ 725.980620] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 725.980629] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 725.980639] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
<br>
[ 725.980652] INFO: task jbd2/xvda5-8:194 blocked for more than
120 seconds.
<br>
[ 725.980663] Not tainted 4.9.0-8-amd64 #1 Debian
4.9.144-3.1
<br>
[ 725.980672] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
<br>
[ 725.980684] jbd2/xvda5-8 D 0 194 2 0x00000080
<br>
[ 725.980693] ffff8800f2dac500 0000000000000000
ffff88000298c380 ffff8800f7e18980
<br>
[ 725.980710] ffff8800f2dfa7c0 ffffc9004093bb30
ffffffff816144b9 ffffc9004093bbe8
<br>
[ 725.980729] 00ffffff81307eaf ffff8800f7e18980
0000000000000000 ffff88000298c380
<br>
[ 725.980746] Call Trace:
<br>
[ 725.980756] [<ffffffff816144b9>] ?
__schedule+0x239/0x6f0
<br>
[ 725.980765] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 725.980774] [<ffffffff816149a2>] ? schedule+0x32/0x80
<br>
[ 725.980783] [<ffffffff81617d4d>] ?
schedule_timeout+0x1dd/0x380
<br>
[ 725.980795] [<ffffffff810f107e>] ?
ktime_get+0x3e/0xb0
<br>
[ 725.980804] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 725.980814] [<ffffffff8161421d>] ?
io_schedule_timeout+0x9d/0x100
<br>
[ 725.980824] [<ffffffff810bcf87>] ?
prepare_to_wait+0x57/0x80
<br>
[ 725.980835] [<ffffffff81615187>] ?
bit_wait_io+0x17/0x60
<br>
[ 725.980844] [<ffffffff81614d45>] ?
__wait_on_bit+0x55/0x80
<br>
[ 725.980853] [<ffffffff81615170>] ? bit_wait+0x50/0x50
<br>
[ 725.980862] [<ffffffff81614eae>] ?
out_of_line_wait_on_bit+0x7e/0xa0
<br>
[ 725.980872] [<ffffffff810bd3f0>] ?
wake_atomic_t_function+0x60/0x60
<br>
[ 725.980887] [<ffffffffc0076e6e>] ?
jbd2_journal_commit_transaction+0xdde/0x17b0 [jbd2]
<br>
[ 725.980899] [<ffffffff81619364>] ?
__switch_to_asm+0x34/0x70
<br>
[ 725.980909] [<ffffffff81619370>] ?
__switch_to_asm+0x40/0x70
<br>
[ 725.980919] [<ffffffff810a1998>] ?
finish_task_switch+0x78/0x210
<br>
[ 725.980929] [<ffffffff81618df4>] ?
_raw_spin_unlock_irqrestore+0x14/0x20
<br>
[ 725.980941] [<ffffffffc007bc62>] ?
kjournald2+0xc2/0x260 [jbd2]
<br>
[ 725.980951] [<ffffffff810bd350>] ?
prepare_to_wait_event+0xf0/0xf0
<br>
[ 725.980962] [<ffffffffc007bba0>] ?
commit_timeout+0x10/0x10 [jbd2]
<br>
[ 725.980973] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0
<br>
[ 725.980981] [<ffffffff8109a500>] ?
kthread_park+0x60/0x60
<br>
[ 725.980990] [<ffffffff816193f7>] ?
ret_from_fork+0x57/0x70
</font><br>
<br>
<br>
<br>
$ xen-diag gnttab_query_size 0
<br>
domid=0: nr_frames=1, max_nr_frames=256
<br>
<br>
$ xen-diag gnttab_query_size 51
<br>
xc: error: Could not query dom 20's grant size: Internal error
<br>
<br>
<br>
$ xl sched-credit
<br>
Cpupool Pool-0: tslice=30ms ratelimit=1000us migration-delay=0us
<br>
Name ID Weight Cap
<br>
Domain-0 0 512 0
<br>
domu1 51 256 0
<br>
<br>
<br>
If you need more information don't hesitate :)
<br>
<br>
Hervé
<br>
<br>
</p>
<div class="moz-signature">-- <br>
<div style="font-family:Helvetica, Arial, sans-serif;
font-size:0.8em;">
<p style="opacity:0.7;"><b style="font-size:1.3em;">Hervé Moris</b><br>
System Administrator / Backend developer</p>
<img style="width:150px;"
src="http://www.loyaltek.com/img/default/logo.png" alt="">
<p style="opacity:0.7;"><b>Loyaltek</b><br>
Cantersteen 47<br>
1000 - Brussels<br>
Tel : +32 2 808.07.58<br>
Fax : +32 2 706.56.84</p>
<a style="opacity:0.7;" href="http://www.loyaltek.com">www.loyaltek.com</a>
<br>
</div>
</div>
</body>
</html>