Monthly Archives: December 2017

Attempting to Run SQL on Linux Inside Windows Subsystem for Linux

Shawn Melton MVP and dbatools contributor last week had an issue running SQL Server on Linux inside of Windows Subsystem for Linux.

I didn’t want to leave a brother hanging so I spent this morning digging into this a little bit. 

Reproducing the Issue

The first thing I had to do was reproduce the issue. So on my Windows 10 test VM I installed the Windows Subsystem for Linux, steps to do so are here and I installed the Ubuntu app.

Then, I fired up a bash shell using WSL and then I installed SQL Server on Linux for Ubuntu as documented here

Now, I completed the installation of SQL Server on Linux using mssql-conf when that program completes it attempts to start SQL Server on Linux. BOOM! I’m able to reproduce the same error.

Looking at the error, I decided to see if I could run SQL Server on Linux from the shell as the user mssql. This would remove systemd and mssql-conf from the picture. Basically I wanted to see if I could get another, more descriptive, error to pop out. To do that we’ll need to change over to the mssql user with su.

sudo su mssql -

And then change into the working directory for SQL Server on Linux and try to launch SQL Server.

cd /var/opt/mssql/
/opt/mssql/bin/sqlservr

Now, doing that…generates same same error! Here’s the error in a search engine friendly form :)

mssql@DESKTOP:~$ /opt/mssql/bin/sqlservr
This program has encountered a fatal error and cannot continue running.
The following diagnostic information is available:
Reason: 0x00000003
Message: fd != -1
Stacktrace: 00007f818942d4d3 00007f8188de76ba 00007f81863e73dd
Process: 79 - sqlservr
Thread: 80
Instance Id: 50bd6e1b-8f6c-45b3-939d-2338725d8b4a
Crash Id: d38007c0-48c6-4374-9205-5539333138ff
Build stamp: 5fb3474a5f63ad2f4b7eddadad44a086839721f18a66c5fb5d7cfcce25c0f539
This program has encountered a fatal error and cannot continue running.
The following diagnostic information is available:
Reason: 0x00000003
Message: fd != -1
Stacktrace: 00007f818942d6ac 00007f8188de76ba 00007f81863e73dd
Process: 81 - sqlservr
Thread: 83
Instance Id: 50bd6e1b-8f6c-45b3-939d-2338725d8b4a
Crash Id: d38007c0-48c6-4374-9205-5539333138ff
Build stamp: 5fb3474a5f63ad2f4b7eddadad44a086839721f18a66c5fb5d7cfcce25c0f539
*********** PANIC CORE DUMP GENERATION FAILED **********
Attempt to launch handle-crash.sh failed with error 0x0000000C
Aborted (core dumped)

Digging a Little Deeper

So now with the same error output, I decided to give it a cursory pass with strace to see if I could find anything that would put us closer to why SQL Server on Linux won’t start when using Windows Subsystem for Linux.

What you see in the strace output is the parent process creating the child sqlservr process and failing. In the first line of output you can see process 137 clone and return process ID 139. Which is how a parent process creates a child in Linux. Then process 139 tries to perform some setup operations like registering signal actions (rt_sigaction) and their corresponding routines to call when that signal is received by that process.

Now the only error I found in the output is the prctl call which returns invalid argument.This system call is to perform operations on a process.  On my WSL system the option being set PR_SET_PTRACER is for the Yama LSM subsystem which lives in /proc/sys/kernel/yama normally. This doesn’t exist on my Ubuntu WSL installation. I checked my CentOS full VMs and this exists. I checked a full Ubuntu installation and it’s there too.

After the error SQL Server calls tgkill and kills itself with the SIGABRT signal. A dump occurs and the program exits. 

137 clone(child_stack=0x7fb1a0feff30, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fb1a0ff09d0, tls=0x7fb1a0ff0700, child_tidptr=0x
7fb1a0ff09d0) = 139
139 set_robust_list(0x7fb1a0ff09e0, 24 <unfinished ...>
137 fcntl(1, F_SETFL, O_RDONLY|O_APPEND <unfinished ...>
139 <... set_robust_list resumed> ) = 0
137 <... fcntl resumed> ) = 0
139 gettid( <unfinished ...>
137 fcntl(2, F_SETFL, O_RDONLY|O_APPEND <unfinished ...>
139 <... gettid resumed> ) = 139
137 <... fcntl resumed> ) = 0
139 rt_sigaction(SIGABRT, {0x7fb1a6a2e470, [ABRT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, <unfinished ...>
137 getrlimit(RLIMIT_NOFILE, <unfinished ...>
139 <... rt_sigaction resumed> {0x7fb1a6a2d290, [ABRT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
137 <... getrlimit resumed> {rlim_cur=1024, rlim_max=4*1024}) = 0
139 rt_sigaction(SIGILL, {0x7fb1a6a2e470, [ILL], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, <unfinished ...>
137 setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024} <unfinished ...>
139 <... rt_sigaction resumed> {0x7fb1a6a52790, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_SIGINFO, 0x7fb1a63f1390}, 8) = 0
137 <... setrlimit resumed> ) = 0
139 rt_sigaction(SIGFPE, {0x7fb1a6a2e470, [FPE], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, <unfinished ...>
137 gettid( <unfinished ...>
139 <... rt_sigaction resumed> {0x7fb1a6a52790, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_SIGINFO, 0x7fb1a63f1390}, 8) = 0
137 <... gettid resumed> ) = 137
139 rt_sigaction(SIGSEGV, {0x7fb1a6a2e470, [SEGV], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, <unfinished ...>
137 rt_sigprocmask(SIG_BLOCK, [TERM], <unfinished ...>
139 <... rt_sigaction resumed> {0x7fb1a6a52790, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_SIGINFO, 0x7fb1a63f1390}, 8) = 0
137 <... rt_sigprocmask resumed> NULL, 8) = 0
139 rt_sigaction(SIGBUS, {0x7fb1a6a2e470, [BUS], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, <unfinished ...>
137 rt_sigtimedwait([TERM], NULL, NULL, 8 <unfinished ...>
139 <... rt_sigaction resumed> {0x7fb1a6a52790, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_SIGINFO, 0x7fb1a63f1390}, 8) = 0
139 rt_sigaction(SIGTRAP, {0x7fb1a6a2e470, [TRAP], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a52790, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_SIGINFO, 0x7fb1a63f1390}, 8) = 0
139 rt_sigaction(SIGSYS, {0x7fb1a6a2e470, [SYS], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2d290, [SYS], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGXCPU, {0x7fb1a6a2e470, [XCPU], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2d290, [XCPU], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGXFSZ, {0x7fb1a6a2e470, [XFSZ], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2d290, [XFSZ], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGSTKFLT, {0x7fb1a6a2e470, [STKFLT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2d290, [STKFLT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb1a03f0000
139 munmap(0x7fb1a03f0000, 4194304) = 0
139 mmap(NULL, 8384512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb19fff0000
139 munmap(0x7fb19fff0000, 65536) = 0
139 munmap(0x7fb1a0400000, 4124672) = 0
139 open("/proc/self/status", O_RDONLY) = 41
139 fstat(41, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
139 read(41, "Name:\tsqlservr\nState:\tS (sleepin"..., 4096) = 663
139 close(41) = 0
139 rt_sigaction(SIGABRT, {0x7fb1a6a2d290, [ABRT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [ABRT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGILL, {0x7fb1a6a2d290, [ILL], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [ILL], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGFPE, {0x7fb1a6a2d290, [FPE], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [FPE], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGSEGV, {0x7fb1a6a2d290, [SEGV], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [SEGV], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGBUS, {0x7fb1a6a2d290, [BUS], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [BUS], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGTRAP, {0x7fb1a6a2d290, [TRAP], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [TRAP], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGSYS, {0x7fb1a6a2d290, [SYS], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [SYS], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGXCPU, {0x7fb1a6a2d290, [XCPU], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [XCPU], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGXFSZ, {0x7fb1a6a2d290, [XFSZ], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [XFSZ], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 rt_sigaction(SIGSTKFLT, {0x7fb1a6a2d290, [STKFLT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, {0x7fb1a6a2e470, [STKFLT], SA_RESTORER|SA_RESTART, 0x7fb1a39154b0}, 8) = 0
139 prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY) = -1 EINVAL (Invalid argument)
139 prctl(PR_SET_PDEATHSIG, SIG_0) = 0
139 open("/proc/self/status", O_RDONLY) = 41
139 fstat(41, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
139 read(41, "Name:\tsqlservr\nState:\tS (sleepin"..., 4096) = 663
139 close(41) = 0
139 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
139 tgkill(137, 139, SIGABRT) = 0
139 --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=137, si_uid=999} ---
139 gettid() = 139
139 write(2, "Dump collecting thread [139] hit"..., 57) = 57
139 exit_group(-1) = ?
137 +++ exited with 255 +++
138 +++ exited with 255 +++
139 +++ exited with 255 +++

What’s Really Happening?

Well I think something is missing from Windows Subsystem for Linux. Is it the Yama stuff…perhaps. But clearly SQL Server isn’t happy with the environment and kills itself. I haven’t dove into WSL yet and I don’t know how it’s implemented, but there could also be something up at that level too. Generally I don’t write blog posts where I don’t know exactly what’s going on, but I did want to let folks know that SQL on Linux doesn’t work on Windows Subsystem for Linux. 


A Novel Idea for High Availability in SQL Server on Linux

Over the past year we’ve learned about how SQL Server on Linux is implemented, leveraging SQLPAL and the team is pretty confident in their architectural decisions as indicated in this post here.

Now that there is this wrapper around SQL Server, this really opens up some interesting opportunities…perhaps we can leverage SQLPAL to facilitate some new high availability techniques.

When I was in graduate school, I worked on a research project, that became my master’s thesis. In this work, I developed a technique that synchronized the process address space of a virtual machine on two separate physical hypervisors.The technique involved an initial copy of all pages between the two systems and then selectively copying the virtual machine’s pages as they became dirty. Using this technique, the process address space of the virtual machine is synchronized between the two hypervisors. This allows for a significant reduction in the amount of information that had to be replicated between the hypervisors but more importantly…the virtual machines memory in sync which meant if hypervisor hosting the virtual machine crashed we could theoretically start the virtual machine on the second hypervisor.

Now, during my PASS Summit talk this year, I presented to the audience my theory that SQLPAL is virtualization. But it’s not machine virtualization, it’s process virtualization. Which means there’s a purpose built environment hosting the SQL Server process. This environment, SQLPAL, is the main allocator of resources from the physical system. It’s the thing that asks for memory, disk, network anything that’s needed from the underlying operating system.

Now, what if we took these two ideas and brought them together? What if SQLPAL was able to synchronize the program state and resources between two separate systems? Could we provide highly available SQL Services with a technique like this? I think we can. Perhaps we don’t even synchronize the pages between the system. Perhaps an even lighter technique could be used, such as duplicating the system calls between the two copies of SQL Server and thus implicitly synchronizing the program state.

Think about the possibilities…we could have a system that fails over with all the context of the currently active system, active connections could stay active, buffer pool populated, plan cache could still exist and not have to be rebuilt. Yes, we’ll likely need some sort of low latency, high bandwidth interconnect..but we have those. And there’s certainly more implementation details that need to be thought through…but I think there’s something here. 

A couple questions I thought of while writing this…

1. Does this provide more value than Availability Groups? I think so…program state remains in sync between the two systems. So things like user connections could be maintained during failover (with the appropriate relocation of the IP of course). I also think the quorum model would be simpler, as there is only one pair in the synchronization.

2. Does this provide more value than virtual machine migration, perhaps. This technique could be hypervisor independent.

I’d love to hear your thoughts on this! Most of all I want you to start thinking about new ways we can leverage SQLPAL and it’s abstraction from hardware.