Given a data lake of tabular data as well as a query table, how can we retrieve all the tables in the data lake that can be unioned with the query table? Table union search constitutes an essential task in data discovery and preparation as it enables data scientists to navigate massive open data repositories. Existing methods identify uniability based on column representations (word surface forms or token embeddings) and column relation represented by column representation similarity. However, the semantic similarity obtained between column representations is often insufficient to reveal latent relational features to describe the column relation between pair of columns and not robust to the table noise. To address these issues, in this paper, we propose a multi-stage self-supervised table union search framework called AUTOTUS, which represents column relation as a vector- column relational representation and learn column relational representation in a multi-stage manner that can better describe column relation for table unionability prediction. In particular, the large language model powered contextualized column relation encoder is updated by adaptive clustering and pseudo label classification iteratively so that the better column relational representation can be learned. Moreover, to improve the robustness of the model against table noises, we propose table noise generator to add table noise to the training table data. Experiments on real-world datasets and synthetic test set augmented with table noise show that AUTOTUS achieves 5.2% performance gain over the SOTA baseline.

@en

The frequency and severity of traffic crashes have commonly been used as indicators of crash risk on transport networks. Comprehensive modeling of crash risk should account for both frequency and injury severity—capturing both the extent and intensity of transport risk for designing effective safety improvement programs. Previous research has revealed that crashes are correlated across severity categories because of the combined influence of risk factors, observed or unobserved. Moreover, crashes are the outcomes of a multitude of factors related to roadway design, traffic operations, pavement conditions, driver behavior, human factors, and environmental characteristics, or in more general terms: factors reflect both engineering and non-engineering risk sources. Perhaps not surprisingly, engineering risk sources have dominated the list of variables in the mainstream modeling of crashes whereas non-engineering sources, in particular, behavioral factors, are crucially omitted. It is plausible to assume that crash contributing factors from the same risk source affect crashes in a similar manner, but their influences vary across different risk sources. Conventional crash frequency modeling hypothesizes that the total crash count at any roadway site is well-approximated by a single risk source to which several explanatory variables contribute collaboratively. The conventional formulation is not capable of accounting for variations between risk sources; therefore, is unable to discriminate distinct impacts between engineering variables and non-engineering variables. To address this shortcoming, this study contributes to the development of multivariate multiple risk source regression, a robust modeling technique to model crash frequency and severity simultaneously. The multivariate multiple risk source regression method applied in this study can effectively capture the correlation between severity levels of crash counts while identifyinging the varying effects of crash contributing factors originated from distinct sources. Using crashes on Wisconsin rural two-lane highways, two risk sources – engineering and behavioral – were employed to develop proposed models. The modeling results were compared with a single equation negative binomial (NB) model, and a univariate multiple risk source model. The results show that the multivariate multiple risk source model significantly outperforms the other models in terms of statistical fit across several measures. The study demonstrates a unique approach to explicitly incorporating behavioral factors into crash prediction models while taking crash severity into consideration. More importantly, the parameter estimates provide more insight into the distinct sources of crash risk, which can be used to further inform safety practitioners and guide roadway improvement programs.@en