Start_cpu can fail on port expander.

[canthony]

Starting a CPU on a port expander may fail.

Condition: There is a running CPU on the expander and start_cpu is not executing on it.

(Given a configuration with CPU A not on an expander, and B, C, D sharing an expander, this condition can arise if:

start_cpu is running on A, B is running and C is added, or

start_cpu is running on B, C is running and D is added.)

Assuming the second case (it is the well analyzed one):

Start_cpu configures the SCU port and the port expander

/* Mask interrupts for new processor and enable all controller ports to new processor. */

               call scr_util$set_mask (scs$interrupt_controller, (pdata.controller_port), scs$sys_level);
                                                            /* Allow no interrupts. */
               call scr_util$set_port_enable ((pdata.controller_port), "1"b);
                                                            /* Set port enaobled on all controllers. */
               if pdata.expanded_port then do;              /* Update port expander bits */
                    call scr_util$set_export_enable ((pdata.controller_port), (pdata.expander_port), "1"b);
                    call scr_util$update_export_xipmsk ((pdata.controller_port));
                                                            /* Make mask from pdata.interrupt_cpu's */
               end;

The "set export enable" configures the expander port to forward "connects" on the SCU port to the CPU on the subport. For the "adding CPU D" case, the expander port would be configured to forward to B, C and D.

Next, start_cpu initializes variables in init_processor, and then sends connects to make all other processors suspend normal operations. (It does not send a connect to itself, thus the two different configurations shown above.)

Because of the way that the port expander works, the connect sent to C will also be delivered to D. Depending on the timing of D's response to the connect it may execute code that renders it unresponsive to the interrupt that start_cpu sends.

This failure is reliably reproducible in the multi-threaded emulator, but is sensitive to timing conditions, indicating that this is a "race condition" between start_cpu's sending of connects and setting up for the interrupt with the response of the started CPU to the connect. It is unknown how the h/w would behave in this situation.

Restricting the configuration to a maximum of two CPUs per port expander seems to remediate the situation for the tc_init$start_other_cpus, as the observed behavior is that a newly added CPU executes the next start_cpu cycle, with the effect that none of the CPUs on the expander receive connects (start_cpu does not send a connect to itself) and so the second CPU on the port expander is safely isolated.

Adding a 100ms delay to start_cpu after the connect lock is released also remediates the race; but the mechanism is not well understood, and so is not necessarily a robust solution.

When adding CPUs with rcf, the race condition has not been observed; it is believed that time taken to process each rcf command allows time for the race condition to abate, but again the mechanism is not well understood.